Skip to content

[fix](function) fix error result in split_by_string with utf8 chars #40710

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
HappenLee merged 3 commits into from
Sep 18, 2024
Merged

[fix](function) fix error result in split_by_string with utf8 chars #40710

HappenLee merged 3 commits into from
Sep 18, 2024

Conversation

@Mryange
Copy link
Contributor

@Mryange Mryange commented Sep 12, 2024

Proposed changes

mysql []>select split_by_string('你a好b世c界','');
+-----------------------------------------------------------------------------+
| split_by_string('你a好b世c界', '')                                          |
+-----------------------------------------------------------------------------+
| ["�", "�", "�", "a", "�", "�", "�", "b", "�", "�", "�", "c", "�", "�", "�"]             |
+-----------------------------------------------------------------------------+
1 row in set (0.03 sec)
now
mysql []>select split_by_string('你a好b世c界','');
+---------------------------------------------+
| split_by_string('你a好b世c界', '')          |
+---------------------------------------------+
| ["你", "a", "好", "b", "世", "c", "界"]     |
+---------------------------------------------+

@doris-robot
Copy link

doris-robot commented Sep 12, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Mryange
Copy link
Contributor Author

Mryange commented Sep 12, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 12, 2024

TPC-H: Total hot run time: 40195 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47b8250297a4be311818bbb6ac4b8ca895af33eb, data reload: false

------ Round 1 ----------------------------------
q1	17910	5096	4259	4259
q2	2609	197	191	191
q3	10965	1421	1434	1421
q4	10332	964	1029	964
q5	8571	3254	3229	3229
q6	233	148	150	148
q7	1059	660	628	628
q8	9482	2018	2027	2018
q9	6828	6281	6323	6281
q10	7010	2559	2505	2505
q11	426	248	255	248
q12	404	236	233	233
q13	17758	3045	3038	3038
q14	292	258	263	258
q15	565	502	495	495
q16	518	426	426	426
q17	1003	948	951	948
q18	7588	6852	6796	6796
q19	1379	1247	1238	1238
q20	614	337	334	334
q21	3958	3565	3569	3565
q22	1083	987	972	972
Total cold run time: 110587 ms
Total hot run time: 40195 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4219	4196	4233	4196
q2	336	249	231	231
q3	2916	2889	2919	2889
q4	1975	1980	1956	1956
q5	5442	5420	5431	5420
q6	222	131	134	131
q7	2067	1696	1719	1696
q8	3259	3328	3330	3328
q9	8431	8443	8400	8400
q10	3388	3427	3442	3427
q11	571	472	470	470
q12	765	572	577	572
q13	4091	3062	3066	3062
q14	302	270	261	261
q15	543	503	486	486
q16	516	459	453	453
q17	1770	1724	1712	1712
q18	7959	7476	7531	7476
q19	1729	1724	1710	1710
q20	2051	1789	1808	1789
q21	5585	5457	5493	5457
q22	1098	1021	984	984
Total cold run time: 59235 ms
Total hot run time: 56106 ms

@doris-robot
Copy link

doris-robot commented Sep 12, 2024

TeamCity be ut coverage result:
Function Coverage: 36.89% (9454/25630)
Line Coverage: 28.24% (77753/275289)
Region Coverage: 27.65% (40146/145193)
Branch Coverage: 24.26% (20394/84080)
Coverage Report: http://coverage.selectdb-in.cc/coverage/47b8250297a4be311818bbb6ac4b8ca895af33eb_47b8250297a4be311818bbb6ac4b8ca895af33eb/report/index.html

@doris-robot
Copy link

doris-robot commented Sep 12, 2024

TPC-DS: Total hot run time: 194681 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 47b8250297a4be311818bbb6ac4b8ca895af33eb, data reload: false

query1	911	377	407	377
query2	6492	1739	1736	1736
query3	6653	212	221	212
query4	28007	24114	23712	23712
query5	5363	557	524	524
query6	280	183	169	169
query7	4590	306	308	306
query8	274	217	229	217
query9	8524	2442	2469	2442
query10	485	295	277	277
query11	16168	15655	15621	15621
query12	160	103	101	101
query13	1675	414	379	379
query14	11365	6985	7435	6985
query15	215	174	177	174
query16	6896	467	484	467
query17	1537	591	585	585
query18	1179	293	295	293
query19	202	154	155	154
query20	123	113	114	113
query21	220	106	106	106
query22	4579	4216	4237	4216
query23	34348	33710	33623	33623
query24	9858	3229	3092	3092
query25	677	415	415	415
query26	1248	159	161	159
query27	2572	281	291	281
query28	6466	2028	2036	2028
query29	926	440	435	435
query30	300	163	155	155
query31	1001	791	786	786
query32	106	57	58	57
query33	741	312	311	311
query34	913	512	486	486
query35	894	758	759	758
query36	1071	922	900	900
query37	163	84	88	84
query38	4019	4067	3897	3897
query39	1468	1431	1395	1395
query40	261	118	117	117
query41	49	47	47	47
query42	121	99	101	99
query43	504	439	430	430
query44	1238	796	755	755
query45	201	174	170	170
query46	1123	816	825	816
query47	1892	1761	1789	1761
query48	374	299	298	298
query49	1143	474	453	453
query50	940	455	464	455
query51	7066	6866	6873	6866
query52	110	92	89	89
query53	265	193	197	193
query54	773	464	481	464
query55	81	76	81	76
query56	286	289	283	283
query57	1215	1071	1074	1071
query58	261	238	249	238
query59	2766	2758	2599	2599
query60	319	304	296	296
query61	149	230	96	96
query62	897	686	657	657
query63	249	201	187	187
query64	5253	683	657	657
query65	3320	3176	3178	3176
query66	1291	337	347	337
query67	15940	15526	15247	15247
query68	3878	606	582	582
query69	419	281	279	279
query70	1104	1085	1071	1071
query71	343	277	287	277
query72	6205	4068	4051	4051
query73	769	339	328	328
query74	9468	9175	8931	8931
query75	3375	2717	2760	2717
query76	2572	1351	1346	1346
query77	447	327	329	327
query78	9826	9305	9632	9305
query79	1131	906	874	874
query80	987	836	825	825
query81	526	272	265	265
query82	886	267	260	260
query83	215	190	198	190
query84	235	111	108	108
query85	736	469	417	417
query86	472	312	310	310
query87	4546	4385	4346	4346
query88	4095	4037	4021	4021
query89	382	370	369	369
query90	1761	316	316	316
query91	125	123	125	123
query92	80	77	76	76
query93	1087	1067	1061	1061
query94	822	393	375	375
query95	511	440	425	425
query96	468	473	465	465
query97	3140	3137	3174	3137
query98	230	227	224	224
query99	1569	1337	1319	1319
Total cold run time: 283166 ms
Total hot run time: 194681 ms

@doris-robot
Copy link

doris-robot commented Sep 12, 2024

ClickBench: Total hot run time: 31.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 47b8250297a4be311818bbb6ac4b8ca895af33eb, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.50	0.50
query6	1.15	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.56	0.52	0.51
query10	0.57	0.59	0.58
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.61	0.60
query14	1.44	1.48	1.48
query15	0.91	0.90	0.89
query16	0.36	0.36	0.36
query17	1.04	1.04	1.02
query18	0.21	0.20	0.20
query19	1.92	1.82	1.80
query20	0.01	0.01	0.01
query21	15.40	0.68	0.66
query22	3.82	8.37	1.26
query23	17.85	1.44	1.46
query24	2.27	0.22	0.22
query25	0.19	0.09	0.09
query26	0.30	0.18	0.19
query27	0.07	0.08	0.07
query28	13.16	1.02	0.98
query29	12.58	3.37	3.37
query30	0.25	0.06	0.06
query31	2.86	0.42	0.41
query32	3.23	0.51	0.50
query33	3.04	3.02	3.04
query34	15.44	4.35	4.35
query35	4.33	4.31	4.35
query36	0.69	0.50	0.49
query37	0.19	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.14	0.13
query41	0.10	0.06	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 107.91 s
Total hot run time: 31.18 s

@Mryange
Copy link
Contributor Author

Mryange commented Sep 14, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 14, 2024

TeamCity be ut coverage result:
Function Coverage: 36.91% (9473/25664)
Line Coverage: 28.28% (77902/275482)
Region Coverage: 27.68% (40228/145323)
Branch Coverage: 24.28% (20436/84152)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4d09fdb820071bce4e87e91b39830fac4fa7a022_4d09fdb820071bce4e87e91b39830fac4fa7a022/report/index.html

@doris-robot
Copy link

doris-robot commented Sep 14, 2024

TPC-H: Total hot run time: 42634 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4d09fdb820071bce4e87e91b39830fac4fa7a022, data reload: false

------ Round 1 ----------------------------------
q1	17563	7236	7217	7217
q2	2042	189	182	182
q3	10769	1230	1406	1230
q4	10349	985	1054	985
q5	7724	3151	3130	3130
q6	240	155	151	151
q7	1028	649	616	616
q8	9444	2008	2021	2008
q9	6745	6281	6251	6251
q10	7022	2505	2526	2505
q11	436	255	250	250
q12	408	222	224	222
q13	17748	3029	3033	3029
q14	282	251	251	251
q15	590	528	522	522
q16	533	431	431	431
q17	998	939	965	939
q18	7313	6858	6687	6687
q19	1377	1236	1227	1227
q20	602	325	326	325
q21	3885	3567	3486	3486
q22	1061	999	990	990
Total cold run time: 108159 ms
Total hot run time: 42634 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7161	7176	7100	7100
q2	339	235	239	235
q3	3248	3121	3075	3075
q4	2060	2097	2066	2066
q5	5671	5589	5685	5589
q6	240	149	153	149
q7	2208	1756	1771	1756
q8	3398	3423	3399	3399
q9	8805	8912	8775	8775
q10	3518	3550	3546	3546
q11	598	498	472	472
q12	819	639	629	629
q13	9274	3234	3183	3183
q14	306	281	278	278
q15	589	549	515	515
q16	521	470	466	466
q17	1788	1778	1750	1750
q18	8385	8167	8146	8146
q19	1768	1742	1741	1741
q20	2104	1869	1864	1864
q21	5799	5406	5680	5406
q22	1127	1040	1024	1024
Total cold run time: 69726 ms
Total hot run time: 61164 ms

@doris-robot
Copy link

doris-robot commented Sep 14, 2024

TPC-DS: Total hot run time: 199937 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4d09fdb820071bce4e87e91b39830fac4fa7a022, data reload: false

query1	1244	894	852	852
query2	6345	1795	1791	1791
query3	10680	3815	3831	3815
query4	56049	25770	24127	24127
query5	5028	555	549	549
query6	326	170	169	169
query7	5633	307	304	304
query8	285	229	220	220
query9	6068	2577	2573	2573
query10	426	300	271	271
query11	15971	15495	15578	15495
query12	150	100	100	100
query13	1434	406	381	381
query14	10516	7345	7383	7345
query15	209	182	193	182
query16	7019	510	475	475
query17	1183	610	622	610
query18	1943	310	311	310
query19	204	161	159	159
query20	172	113	111	111
query21	210	104	106	104
query22	4628	4652	4539	4539
query23	34673	33657	33675	33657
query24	6149	3029	3020	3020
query25	502	381	394	381
query26	603	149	156	149
query27	1579	279	286	279
query28	2814	2117	2068	2068
query29	676	407	416	407
query30	220	155	150	150
query31	972	775	790	775
query32	72	53	53	53
query33	454	285	293	285
query34	889	475	479	475
query35	844	714	725	714
query36	1067	885	884	884
query37	141	81	81	81
query38	3997	3943	4002	3943
query39	1450	1442	1378	1378
query40	208	112	112	112
query41	47	46	45	45
query42	118	101	99	99
query43	489	448	447	447
query44	1177	771	766	766
query45	194	169	173	169
query46	1092	810	793	793
query47	1912	1753	1816	1753
query48	375	293	324	293
query49	733	473	456	456
query50	927	437	444	437
query51	7172	6845	6936	6845
query52	97	86	89	86
query53	252	177	177	177
query54	602	469	489	469
query55	75	79	78	78
query56	300	272	260	260
query57	1193	1092	1065	1065
query58	231	277	318	277
query59	2932	2569	2731	2569
query60	298	271	272	271
query61	105	101	98	98
query62	819	666	670	666
query63	217	195	190	190
query64	1592	731	645	645
query65	3212	3156	3179	3156
query66	671	292	302	292
query67	15919	15551	15369	15369
query68	2011	587	599	587
query69	400	285	292	285
query70	1147	1121	1099	1099
query71	337	287	288	287
query72	4648	4068	4075	4068
query73	763	325	329	325
query74	9269	9262	9033	9033
query75	3349	2707	2739	2707
query76	1356	1299	1299	1299
query77	482	320	320	320
query78	9918	9396	9260	9260
query79	935	902	881	881
query80	837	886	811	811
query81	463	267	262	262
query82	263	268	262	262
query83	197	183	190	183
query84	262	107	108	107
query85	703	405	401	401
query86	316	305	309	305
query87	4411	4420	4497	4420
query88	4190	4111	4095	4095
query89	377	365	369	365
query90	1947	308	357	308
query91	123	126	128	126
query92	78	76	77	76
query93	1063	1059	1060	1059
query94	755	385	390	385
query95	460	414	421	414
query96	470	470	471	470
query97	3143	3138	3132	3132
query98	229	226	222	222
query99	1715	1319	1326	1319
Total cold run time: 292401 ms
Total hot run time: 199937 ms

@doris-robot
Copy link

doris-robot commented Sep 14, 2024

ClickBench: Total hot run time: 30.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4d09fdb820071bce4e87e91b39830fac4fa7a022, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.05	0.04
query3	0.22	0.04	0.04
query4	1.68	0.06	0.07
query5	0.49	0.51	0.50
query6	1.12	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.57	0.52	0.50
query10	0.56	0.58	0.56
query11	0.15	0.11	0.11
query12	0.15	0.13	0.13
query13	0.62	0.61	0.61
query14	1.46	1.46	1.48
query15	0.89	0.89	0.89
query16	0.36	0.36	0.36
query17	1.04	1.03	1.00
query18	0.17	0.16	0.17
query19	1.96	1.86	1.82
query20	0.01	0.01	0.02
query21	15.39	0.66	0.65
query22	4.19	8.41	1.27
query23	17.84	1.32	1.37
query24	2.26	0.22	0.23
query25	0.19	0.09	0.09
query26	0.30	0.18	0.18
query27	0.08	0.07	0.08
query28	13.19	1.12	1.09
query29	12.57	3.34	3.30
query30	0.25	0.05	0.06
query31	2.88	0.41	0.42
query32	3.22	0.50	0.49
query33	2.99	3.04	3.07
query34	15.49	4.28	4.30
query35	4.35	4.34	4.36
query36	0.69	0.49	0.48
query37	0.18	0.16	0.17
query38	0.18	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.09	0.06	0.04
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 108.29 s
Total hot run time: 30.95 s

HappenLee
HappenLee reviewed Sep 18, 2024
View reviewed changes
@Mryange Mryange force-pushed the fix-split_by_string-with-utf8 branch from 4d09fdb to 8251f74 Compare September 18, 2024 01:45
@Mryange
Copy link
Contributor Author

Mryange commented Sep 18, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 18, 2024

TeamCity be ut coverage result:
Function Coverage: 37.33% (9576/25655)
Line Coverage: 28.71% (79145/275681)
Region Coverage: 28.18% (40975/145386)
Branch Coverage: 24.81% (20881/84180)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8251f740f548d85c695b4839a2041ceae065311c_8251f740f548d85c695b4839a2041ceae065311c/report/index.html

@doris-robot
Copy link

doris-robot commented Sep 18, 2024

TPC-H: Total hot run time: 41605 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8251f740f548d85c695b4839a2041ceae065311c, data reload: false

------ Round 1 ----------------------------------
q1	17637	7400	7348	7348
q2	2046	163	165	163
q3	10616	1110	1205	1110
q4	10553	824	703	703
q5	7769	3091	3043	3043
q6	236	154	156	154
q7	1005	611	612	611
q8	9426	2027	2023	2023
q9	6815	6399	6407	6399
q10	6994	2303	2254	2254
q11	430	240	243	240
q12	411	218	218	218
q13	17776	2975	2959	2959
q14	233	215	224	215
q15	578	528	525	525
q16	695	632	610	610
q17	975	796	755	755
q18	7244	6841	6786	6786
q19	1402	1033	1015	1015
q20	583	298	287	287
q21	4044	3183	3170	3170
q22	1140	1017	1029	1017
Total cold run time: 108608 ms
Total hot run time: 41605 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7289	7273	7270	7270
q2	322	231	230	230
q3	3032	2948	2952	2948
q4	2035	1851	1878	1851
q5	5572	5567	5669	5567
q6	239	153	149	149
q7	2187	1797	1781	1781
q8	3317	3402	3425	3402
q9	8671	8889	8715	8715
q10	3513	3389	3515	3389
q11	577	499	485	485
q12	804	648	603	603
q13	10409	3177	3084	3084
q14	311	273	283	273
q15	573	537	534	534
q16	693	675	659	659
q17	1807	1578	1565	1565
q18	8171	7788	7802	7788
q19	1775	1616	1694	1616
q20	2130	1878	1893	1878
q21	5554	5249	5340	5249
q22	1153	1053	1044	1044
Total cold run time: 70134 ms
Total hot run time: 60080 ms

@doris-robot
Copy link

doris-robot commented Sep 18, 2024

TPC-DS: Total hot run time: 199234 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8251f740f548d85c695b4839a2041ceae065311c, data reload: false

query1	1278	853	877	853
query2	6415	2046	2058	2046
query3	10768	4009	3725	3725
query4	64549	29171	23526	23526
query5	5024	462	469	462
query6	420	184	160	160
query7	5468	307	303	303
query8	313	227	230	227
query9	8510	2646	2635	2635
query10	475	298	293	293
query11	17459	15085	15708	15085
query12	161	101	106	101
query13	1464	443	415	415
query14	10767	7530	7601	7530
query15	216	175	182	175
query16	6593	484	480	480
query17	1212	588	574	574
query18	1339	304	310	304
query19	208	144	146	144
query20	125	112	109	109
query21	210	100	101	100
query22	4903	4489	4953	4489
query23	35014	33774	33909	33774
query24	5980	2916	2843	2843
query25	502	387	387	387
query26	609	158	153	153
query27	1592	280	279	279
query28	4002	2444	2403	2403
query29	666	428	415	415
query30	230	149	152	149
query31	972	771	786	771
query32	69	51	52	51
query33	388	283	284	283
query34	885	495	477	477
query35	845	718	713	713
query36	1054	944	901	901
query37	141	88	92	88
query38	3980	3963	3903	3903
query39	1468	1408	1398	1398
query40	198	96	94	94
query41	48	47	46	46
query42	115	99	93	93
query43	535	481	470	470
query44	1123	815	792	792
query45	192	162	167	162
query46	1119	771	729	729
query47	1945	1795	1835	1795
query48	475	362	359	359
query49	669	391	388	388
query50	821	397	403	397
query51	7098	6919	6873	6873
query52	93	86	86	86
query53	248	174	174	174
query54	551	451	448	448
query55	76	72	76	72
query56	284	266	256	256
query57	1239	1093	1096	1093
query58	227	238	231	231
query59	3279	3299	3084	3084
query60	307	271	274	271
query61	131	125	164	125
query62	773	656	654	654
query63	216	191	183	183
query64	1271	657	639	639
query65	3225	3165	3174	3165
query66	643	295	300	295
query67	16046	15646	15583	15583
query68	1245	846	888	846
query69	451	339	351	339
query70	1149	1184	1186	1184
query71	322	333	323	323
query72	4595	3451	3457	3451
query73	578	576	573	573
query74	9207	8984	9003	8984
query75	2938	2845	2956	2845
query76	947	846	862	846
query77	462	359	373	359
query78	9703	9354	9297	9297
query79	884	877	860	860
query80	557	566	568	566
query81	450	252	252	252
query82	238	224	229	224
query83	160	164	164	164
query84	209	110	103	103
query85	653	370	378	370
query86	306	326	306	306
query87	4380	4414	4436	4414
query88	4175	4071	4000	4000
query89	367	359	356	356
query90	1230	306	302	302
query91	165	169	162	162
query92	77	73	71	71
query93	876	874	875	874
query94	519	366	367	366
query95	430	410	408	408
query96	479	479	477	477
query97	3125	3182	3119	3119
query98	219	222	234	222
query99	1317	1290	1309	1290
Total cold run time: 301634 ms
Total hot run time: 199234 ms

@doris-robot
Copy link

doris-robot commented Sep 18, 2024

ClickBench: Total hot run time: 32.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8251f740f548d85c695b4839a2041ceae065311c, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.65	0.10	0.10
query5	0.52	0.51	0.50
query6	1.14	0.74	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.56	0.56	0.55
query11	0.14	0.12	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.60
query14	3.09	2.98	3.00
query15	0.88	0.82	0.82
query16	0.36	0.37	0.38
query17	0.98	1.06	0.98
query18	0.19	0.19	0.19
query19	1.95	1.91	2.00
query20	0.01	0.01	0.00
query21	15.36	0.58	0.58
query22	2.26	2.79	2.04
query23	17.60	0.85	0.74
query24	2.55	0.46	1.43
query25	0.19	0.11	0.10
query26	0.41	0.14	0.13
query27	0.03	0.04	0.04
query28	11.10	1.10	1.07
query29	12.55	3.27	3.26
query30	0.24	0.06	0.06
query31	2.88	0.39	0.38
query32	3.27	0.45	0.47
query33	2.98	2.97	3.01
query34	17.09	4.38	4.39
query35	4.41	4.46	4.44
query36	0.65	0.49	0.49
query37	0.08	0.05	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.03
query40	0.16	0.11	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.21 s
Total hot run time: 32.46 s

@Mryange Mryange force-pushed the fix-split_by_string-with-utf8 branch from 8251f74 to 0baf480 Compare September 18, 2024 02:49
@Mryange
Copy link
Contributor Author

Mryange commented Sep 18, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 18, 2024

TPC-H: Total hot run time: 41673 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0baf4808ad10be06a7c7168d2a6eb0fafeedd5a3, data reload: false

------ Round 1 ----------------------------------
q1	18455	7487	7384	7384
q2	2509	173	175	173
q3	11836	1156	1207	1156
q4	10752	746	823	746
q5	7951	3230	3138	3138
q6	239	146	149	146
q7	1009	611	617	611
q8	9422	2080	2012	2012
q9	6845	6458	6426	6426
q10	7001	2288	2318	2288
q11	435	248	253	248
q12	404	218	214	214
q13	17811	2966	2995	2966
q14	252	207	217	207
q15	567	530	527	527
q16	664	606	612	606
q17	993	838	827	827
q18	7353	6589	6648	6589
q19	1400	1036	1041	1036
q20	584	281	292	281
q21	3952	3082	3251	3082
q22	1123	1010	1031	1010
Total cold run time: 111557 ms
Total hot run time: 41673 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7282	7245	7216	7216
q2	323	230	224	224
q3	2919	2812	2801	2801
q4	1930	1687	1695	1687
q5	5368	5421	5459	5421
q6	224	135	139	135
q7	2086	1749	1691	1691
q8	3209	3353	3337	3337
q9	8463	8496	8454	8454
q10	3370	3362	3356	3356
q11	572	477	477	477
q12	790	589	575	575
q13	5615	2972	2985	2972
q14	293	252	268	252
q15	549	522	511	511
q16	707	661	656	656
q17	1778	1568	1556	1556
q18	7828	7458	7438	7438
q19	1697	1614	1503	1503
q20	2053	1815	1858	1815
q21	5323	5076	5061	5061
q22	1124	1025	1026	1025
Total cold run time: 63503 ms
Total hot run time: 58163 ms

HappenLee
HappenLee approved these changes Sep 18, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 18, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 18, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Sep 18, 2024

TPC-DS: Total hot run time: 194326 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0baf4808ad10be06a7c7168d2a6eb0fafeedd5a3, data reload: false

query1	944	380	384	380
query2	6550	2101	2042	2042
query3	6708	217	227	217
query4	34259	23425	23433	23425
query5	4356	501	467	467
query6	270	178	173	173
query7	4625	299	314	299
query8	316	233	228	228
query9	9736	2659	2642	2642
query10	497	277	274	274
query11	18267	15272	15205	15205
query12	168	105	98	98
query13	1629	416	402	402
query14	10830	7611	7070	7070
query15	306	171	178	171
query16	7355	493	467	467
query17	1749	554	546	546
query18	1985	293	299	293
query19	362	144	141	141
query20	116	107	112	107
query21	216	104	100	100
query22	4467	4358	4239	4239
query23	34799	34067	33950	33950
query24	11158	2968	2796	2796
query25	637	382	392	382
query26	1242	160	155	155
query27	2345	279	278	278
query28	7994	2482	2428	2428
query29	797	408	398	398
query30	319	161	163	161
query31	984	761	811	761
query32	102	53	55	53
query33	768	303	290	290
query34	957	470	487	470
query35	852	758	726	726
query36	1053	926	940	926
query37	170	90	86	86
query38	4058	3988	3993	3988
query39	1491	1549	1401	1401
query40	212	95	94	94
query41	50	49	50	49
query42	115	98	100	98
query43	519	470	469	469
query44	1238	829	788	788
query45	193	195	165	165
query46	1133	761	774	761
query47	1896	1794	1813	1794
query48	440	350	362	350
query49	1117	428	407	407
query50	833	400	401	400
query51	6990	7030	6794	6794
query52	107	89	88	88
query53	250	179	182	179
query54	1278	451	455	451
query55	77	80	83	80
query56	289	266	267	266
query57	1208	1068	1063	1063
query58	243	227	253	227
query59	3009	2848	2797	2797
query60	331	270	267	267
query61	103	101	105	101
query62	873	651	663	651
query63	214	186	186	186
query64	5178	649	634	634
query65	3242	3186	3199	3186
query66	1434	308	299	299
query67	16118	15742	15482	15482
query68	3500	848	841	841
query69	474	346	336	336
query70	1225	1166	1203	1166
query71	333	331	331	331
query72	6130	3489	3388	3388
query73	588	592	583	583
query74	9875	8989	8968	8968
query75	3056	2889	2852	2852
query76	2059	864	858	858
query77	406	365	366	365
query78	9453	9219	9484	9219
query79	922	879	857	857
query80	610	572	574	572
query81	448	253	251	251
query82	237	233	232	232
query83	165	155	162	155
query84	232	111	101	101
query85	692	371	358	358
query86	299	315	315	315
query87	4366	4346	4402	4346
query88	4587	4077	4070	4070
query89	379	361	369	361
query90	1477	321	321	321
query91	175	178	179	178
query92	81	74	77	74
query93	913	915	892	892
query94	583	383	376	376
query95	430	420	416	416
query96	500	498	495	495
query97	3167	3171	3191	3171
query98	233	233	239	233
query99	1424	1284	1289	1284
Total cold run time: 293146 ms
Total hot run time: 194326 ms

@doris-robot
Copy link

doris-robot commented Sep 18, 2024

ClickBench: Total hot run time: 32.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0baf4808ad10be06a7c7168d2a6eb0fafeedd5a3, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.64	0.10	0.11
query5	0.49	0.50	0.51
query6	1.12	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.55	0.49	0.49
query10	0.53	0.55	0.54
query11	0.14	0.11	0.11
query12	0.13	0.11	0.11
query13	0.60	0.59	0.59
query14	2.98	2.97	2.98
query15	0.87	0.82	0.82
query16	0.37	0.38	0.38
query17	1.01	1.00	1.06
query18	0.23	0.20	0.20
query19	1.94	1.92	1.93
query20	0.01	0.00	0.00
query21	15.37	0.60	0.58
query22	2.61	2.01	1.48
query23	17.16	0.97	0.79
query24	2.54	1.34	1.28
query25	0.27	0.14	0.05
query26	0.37	0.14	0.14
query27	0.04	0.04	0.05
query28	10.54	1.12	1.08
query29	12.54	3.34	3.32
query30	0.25	0.06	0.06
query31	2.86	0.38	0.38
query32	3.29	0.46	0.44
query33	2.96	3.05	3.05
query34	17.08	4.38	4.35
query35	4.44	4.45	4.42
query36	0.68	0.47	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.17	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.49 s
Total hot run time: 32.86 s

@doris-robot
Copy link

doris-robot commented Sep 18, 2024

TeamCity be ut coverage result:
Function Coverage: 37.32% (9575/25655)
Line Coverage: 28.70% (79117/275704)
Region Coverage: 28.18% (40979/145402)
Branch Coverage: 24.81% (20888/84190)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0baf4808ad10be06a7c7168d2a6eb0fafeedd5a3_0baf4808ad10be06a7c7168d2a6eb0fafeedd5a3/report/index.html

@HappenLee HappenLee merged commit d5c24d3 into apache:master Sep 18, 2024
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@Mryange @dataroaring
[fix](function) fix error result in split_by_string with utf8 chars (#…
7559534 
…40710)

```
mysql []>select split_by_string('你a好b世c界','');
+-----------------------------------------------------------------------------+
| split_by_string('你a好b世c界', '')                                          |
+-----------------------------------------------------------------------------+
| ["�", "�", "�", "a", "�", "�", "�", "b", "�", "�", "�", "c", "�", "�", "�"]             |
+-----------------------------------------------------------------------------+
1 row in set (0.03 sec)
now
mysql []>select split_by_string('你a好b世c界','');
+---------------------------------------------+
| split_by_string('你a好b世c界', '')          |
+---------------------------------------------+
| ["你", "a", "好", "b", "世", "c", "界"]     |
+---------------------------------------------+
```
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Closed
github-actions bot pushed a commit that referenced this pull request May 8, 2025
@Mryange
[fix](function) fix error result in split_by_string with utf8 chars (#…
dfd5b87 
…40710)

```
mysql []>select split_by_string('你a好b世c界','');
+-----------------------------------------------------------------------------+
| split_by_string('你a好b世c界', '')                                          |
+-----------------------------------------------------------------------------+
| ["�", "�", "�", "a", "�", "�", "�", "b", "�", "�", "�", "c", "�", "�", "�"]             |
+-----------------------------------------------------------------------------+
1 row in set (0.03 sec)
now
mysql []>select split_by_string('你a好b世c界','');
+---------------------------------------------+
| split_by_string('你a好b世c界', '')          |
+---------------------------------------------+
| ["你", "a", "好", "b", "世", "c", "界"]     |
+---------------------------------------------+
```
@github-actions github-actions bot mentioned this pull request May 8, 2025
Merged
yiguolei pushed a commit that referenced this pull request May 8, 2025
@github-actions @Mryange
branch-2.1: [fix](function) fix error result in split_by_string with …
1b10860 
…utf8 chars #40710 (#50689)

Cherry-picked from #40710

Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com>
@yiguolei yiguolei mentioned this pull request May 13, 2025
Closed
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Sep 26, 2025
fix(function): fix error result in split_by_string with utf8 chars ap…
2b323bc 
…ache#40710 (apache#50689)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HappenLee HappenLee HappenLee approved these changes

@Gabriel39 Gabriel39 Gabriel39 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Mryange @doris-robot @HappenLee @Gabriel39 @yiguolei @dataroaring