Skip to content

[feat](function) SUBSTRING_INDEX function delimiter supports dynamic #50149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 7 commits into from
Apr 22, 2025
Merged

[feat](function) SUBSTRING_INDEX function delimiter supports dynamic #50149

yiguolei merged 7 commits into from
Apr 22, 2025

Conversation

@felixwluo
Copy link
Contributor

@felixwluo felixwluo commented Apr 17, 2025

What problem does this PR solve?

Problem Summary:

sql

SELECT t1.no
      ,t1.sub_str
      ,t1.str
      ,substring_index(t1.str, t1.sub_str, -1)
      ,t2.rst2
FROM (
       SELECT 1 AS no, 'BBB' AS sub_str, 'AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 2 AS no, 'ccc' AS sub_str, 'zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06' AS str UNION ALL
       SELECT 3 AS no, 'DDD' AS sub_str, 'AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 4 AS no, 'DDD' AS sub_str, 'sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06' AS str UNION ALL
       SELECT 5 AS no, 'eee' AS sub_str, 'cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 6 AS no, 'A_01' AS sub_str, 'AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06' AS str 
     ) t1
LEFT JOIN (
       SELECT 1 AS no, 'BBB' AS sub_str,  substring_index('AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06', 'BBB', -1) AS rst2 UNION ALL
       SELECT 2 AS no, 'ccc' AS sub_str,  substring_index('zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06', 'ccc', -1) AS rst2 UNION ALL
       SELECT 3 AS no, 'DDD' AS sub_str,  substring_index('AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06', 'DDD', -1) AS rst2 UNION ALL
       SELECT 4 AS no, 'DDD' AS sub_str,  substring_index('sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06', 'DDD', -1) AS rst2 UNION ALL
       SELECT 5 AS no, 'eee' AS sub_str,  substring_index('cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06', 'eee', -1) AS rst2 UNION ALL
       SELECT 6 AS no, 'A_01' AS sub_str, substring_index('AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06', 'A_01', -1) AS rst2 
     ) t2
     ON t1.no = t2.no AND t1.sub_str = t2.sub_str
ORDER BY t1.no;

previous results for doris:

-- master
function except for the first argument, other parameter must be a constant

-- 2.1
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+
| no   | sub_str | str                                       | substring_index(`t1`.`str`, `t1`.`sub_str`, -1) | rst2                                      |
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+
|    1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06                 | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|    2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06       | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|    3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06                 | _04|EEE_05|FFF_06                         |
|    4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06       | _04|rfv_05|rgb_06                         |
|    5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06       | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|    6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+

mysql result:

+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| no | sub_str | str                                       | substring_index(t1.str, t1.sub_str, -1)   | rst2                                      |
+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|  1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06           | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|  2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|  3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _04|EEE_05|FFF_06                         | _04|EEE_05|FFF_06                         |
|  4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | _04|rfv_05|rgb_06                         | _04|rfv_05|rgb_06                         |
|  5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|  6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+

current doris results:

+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| no   | sub_str | str                                       | substring_index(t1.str, t1.sub_str, -1)   | rst2                                      |
+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|    1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06           | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|    2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|    3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _04|EEE_05|FFF_06                         | _04|EEE_05|FFF_06                         |
|    4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | _04|rfv_05|rgb_06                         | _04|rfv_05|rgb_06                         |
|    5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|    6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+

now consistent with mysql behavior.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Apr 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@felixwluo
Copy link
Contributor Author

felixwluo commented Apr 17, 2025

run buildall

@felixwluo felixwluo requested a review from yiguolei April 17, 2025 13:59
@doris-robot
Copy link

doris-robot commented Apr 17, 2025

TPC-H: Total hot run time: 33675 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c33ef603619246db87315684957e8f9fea09ff5, data reload: false

------ Round 1 ----------------------------------
q1	25657	5006	5009	5006
q2	2062	300	201	201
q3	10351	1217	689	689
q4	10212	993	536	536
q5	7537	2285	2397	2285
q6	184	168	136	136
q7	903	740	603	603
q8	9311	1266	1110	1110
q9	6842	5106	5087	5087
q10	6867	2317	1889	1889
q11	494	285	268	268
q12	354	349	218	218
q13	17782	3635	3076	3076
q14	217	213	206	206
q15	529	487	481	481
q16	441	446	401	401
q17	585	840	364	364
q18	7653	7092	7007	7007
q19	1593	970	547	547
q20	330	329	214	214
q21	3936	2684	2397	2397
q22	1041	1012	954	954
Total cold run time: 114881 ms
Total hot run time: 33675 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5144	5231	5093	5093
q2	232	328	230	230
q3	2161	2727	2326	2326
q4	1430	1832	1442	1442
q5	4469	4439	4473	4439
q6	228	172	128	128
q7	2011	1921	1743	1743
q8	2586	2615	2528	2528
q9	7216	7225	7031	7031
q10	3012	3195	2757	2757
q11	571	505	495	495
q12	715	744	605	605
q13	3500	3890	3257	3257
q14	293	301	271	271
q15	528	497	479	479
q16	465	525	458	458
q17	1136	1556	1387	1387
q18	7748	7532	7349	7349
q19	804	806	877	806
q20	1983	2051	1829	1829
q21	5163	4771	4750	4750
q22	1094	1010	1024	1010
Total cold run time: 52489 ms
Total hot run time: 50413 ms

@doris-robot
Copy link

doris-robot commented Apr 17, 2025

TPC-DS: Total hot run time: 193725 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c33ef603619246db87315684957e8f9fea09ff5, data reload: false

query1	1429	1114	1070	1070
query2	6135	1794	1786	1786
query3	11029	4427	4434	4427
query4	54456	25021	23579	23579
query5	5102	533	442	442
query6	335	202	195	195
query7	4956	525	287	287
query8	308	240	246	240
query9	5805	2581	2580	2580
query10	423	335	271	271
query11	15181	15131	14843	14843
query12	157	113	98	98
query13	1028	499	383	383
query14	10200	6502	6672	6502
query15	200	191	190	190
query16	7020	669	528	528
query17	1114	735	634	634
query18	1551	414	332	332
query19	199	199	160	160
query20	134	125	120	120
query21	203	135	112	112
query22	4360	4326	4253	4253
query23	34329	33474	33430	33430
query24	6697	2475	2471	2471
query25	469	513	417	417
query26	705	290	159	159
query27	2260	518	344	344
query28	3103	2126	2116	2116
query29	578	597	441	441
query30	278	226	203	203
query31	869	865	782	782
query32	76	62	65	62
query33	471	379	321	321
query34	786	925	538	538
query35	818	830	767	767
query36	937	997	895	895
query37	126	104	79	79
query38	4223	4267	4329	4267
query39	1515	1513	1483	1483
query40	231	127	109	109
query41	53	54	49	49
query42	130	111	102	102
query43	505	520	479	479
query44	1380	817	823	817
query45	189	178	168	168
query46	875	1086	658	658
query47	1849	1903	1781	1781
query48	395	422	298	298
query49	685	537	413	413
query50	729	749	414	414
query51	4267	4294	4267	4267
query52	118	111	99	99
query53	235	272	200	200
query54	592	593	538	538
query55	99	96	90	90
query56	335	343	317	317
query57	1168	1179	1144	1144
query58	282	279	265	265
query59	2590	2681	2655	2655
query60	357	345	331	331
query61	154	151	149	149
query62	753	735	716	716
query63	248	224	193	193
query64	1664	1073	716	716
query65	4481	4403	4237	4237
query66	783	400	298	298
query67	15882	15552	15182	15182
query68	7034	849	514	514
query69	538	305	267	267
query70	1160	1072	1050	1050
query71	513	322	308	308
query72	5795	4883	5144	4883
query73	1223	725	349	349
query74	9155	9137	9099	9099
query75	3852	3222	2704	2704
query76	4238	1224	808	808
query77	617	388	282	282
query78	10033	10188	9387	9387
query79	1977	813	584	584
query80	675	538	441	441
query81	482	259	229	229
query82	446	140	101	101
query83	255	260	237	237
query84	295	109	86	86
query85	791	363	317	317
query86	379	307	282	282
query87	4375	4345	4339	4339
query88	3622	2227	2237	2227
query89	412	315	285	285
query90	1906	219	230	219
query91	144	143	111	111
query92	74	64	56	56
query93	1132	987	589	589
query94	648	425	315	315
query95	373	302	288	288
query96	495	576	274	274
query97	3160	3280	3115	3115
query98	233	220	211	211
query99	1434	1413	1276	1276
Total cold run time: 298060 ms
Total hot run time: 193725 ms

@doris-robot
Copy link

doris-robot commented Apr 17, 2025

ClickBench: Total hot run time: 29.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c33ef603619246db87315684957e8f9fea09ff5, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.21	0.20
query4	1.59	0.20	0.10
query5	0.56	0.54	0.55
query6	1.21	0.71	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.60	0.52	0.50
query10	0.58	0.56	0.57
query11	0.15	0.10	0.11
query12	0.15	0.11	0.11
query13	0.62	0.61	0.61
query14	1.15	1.17	1.20
query15	0.88	0.85	0.86
query16	0.38	0.39	0.39
query17	1.00	1.04	1.02
query18	0.21	0.19	0.19
query19	1.86	1.79	1.77
query20	0.01	0.01	0.02
query21	15.42	0.92	0.55
query22	0.75	1.24	0.66
query23	14.91	1.40	0.61
query24	7.03	1.38	0.52
query25	0.52	0.16	0.06
query26	0.60	0.16	0.13
query27	0.05	0.05	0.05
query28	9.46	0.84	0.42
query29	12.92	4.02	3.30
query30	0.25	0.09	0.06
query31	2.83	0.59	0.39
query32	3.23	0.55	0.47
query33	3.06	3.04	3.06
query34	15.76	5.08	4.48
query35	4.48	4.48	4.55
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.05	0.03
query39	0.03	0.02	0.03
query40	0.16	0.14	0.13
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 103.84 s
Total hot run time: 29.03 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 17, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/29) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.15% (14420/27132)
Line Coverage 42.00% (124941/297453)
Region Coverage 40.82% (63833/156376)
Branch Coverage 35.46% (32105/90532)

@felixwluo
Copy link
Contributor Author

felixwluo commented Apr 17, 2025

run cloud_p0

@yiguolei yiguolei added usercase Important user case type label dev/2.1.x dev/3.0.x labels Apr 18, 2025
@felixwluo
Copy link
Contributor Author

felixwluo commented Apr 18, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 18, 2025

TPC-H: Total hot run time: 34103 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e2a77a25c8f52ce89ce0df3c667801f20bb3421, data reload: false

------ Round 1 ----------------------------------
q1	26197	5053	5287	5053
q2	2069	280	186	186
q3	10390	1263	695	695
q4	10219	1007	540	540
q5	7536	2366	2365	2365
q6	177	168	136	136
q7	929	731	603	603
q8	9305	1273	1158	1158
q9	6964	5085	5096	5085
q10	6814	2291	1890	1890
q11	486	283	264	264
q12	360	346	215	215
q13	17760	3699	3085	3085
q14	232	214	213	213
q15	539	479	487	479
q16	449	458	407	407
q17	605	827	384	384
q18	7521	7132	7148	7132
q19	1592	955	544	544
q20	324	330	212	212
q21	4257	3392	2501	2501
q22	1040	1001	956	956
Total cold run time: 115765 ms
Total hot run time: 34103 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5178	5061	5121	5061
q2	246	327	228	228
q3	2165	2615	2242	2242
q4	1432	1843	1441	1441
q5	4483	4369	4394	4369
q6	216	166	127	127
q7	1982	1915	1760	1760
q8	2645	2623	2526	2526
q9	7265	7171	7109	7109
q10	3013	3173	2741	2741
q11	564	543	493	493
q12	675	772	622	622
q13	3484	3924	3289	3289
q14	289	301	287	287
q15	527	486	483	483
q16	450	517	461	461
q17	1166	1547	1412	1412
q18	7720	7651	7377	7377
q19	814	823	969	823
q20	2030	2008	1821	1821
q21	5227	4803	4753	4753
q22	1087	1080	1008	1008
Total cold run time: 52658 ms
Total hot run time: 50433 ms

@doris-robot
Copy link

doris-robot commented Apr 18, 2025

TPC-DS: Total hot run time: 192562 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1e2a77a25c8f52ce89ce0df3c667801f20bb3421, data reload: false

query1	1385	1094	1076	1076
query2	6105	1791	1772	1772
query3	11051	4452	4556	4452
query4	52479	26001	23438	23438
query5	5098	597	461	461
query6	333	204	207	204
query7	4875	503	291	291
query8	317	246	240	240
query9	5501	2549	2542	2542
query10	447	323	267	267
query11	15115	15038	14776	14776
query12	161	121	105	105
query13	1029	519	386	386
query14	10193	6224	6431	6224
query15	215	231	170	170
query16	7094	673	499	499
query17	1074	743	572	572
query18	1572	412	320	320
query19	191	193	177	177
query20	141	124	118	118
query21	205	126	104	104
query22	4529	4523	4210	4210
query23	34177	33405	33214	33214
query24	6602	2442	2452	2442
query25	456	453	416	416
query26	734	276	156	156
query27	2320	519	336	336
query28	3007	2118	2091	2091
query29	582	562	430	430
query30	264	219	190	190
query31	894	846	788	788
query32	75	70	61	61
query33	476	377	315	315
query34	988	859	516	516
query35	772	836	799	799
query36	941	1006	892	892
query37	121	103	76	76
query38	4165	4256	4307	4256
query39	1522	1479	1434	1434
query40	216	124	109	109
query41	54	52	55	52
query42	127	104	106	104
query43	493	522	480	480
query44	1378	832	818	818
query45	184	180	176	176
query46	875	1030	647	647
query47	1864	1873	1790	1790
query48	413	431	309	309
query49	677	517	413	413
query50	705	706	423	423
query51	4282	4397	4224	4224
query52	107	112	101	101
query53	254	273	200	200
query54	602	583	503	503
query55	82	80	85	80
query56	288	323	300	300
query57	1212	1189	1164	1164
query58	272	260	263	260
query59	2636	2796	2666	2666
query60	352	347	303	303
query61	133	133	134	133
query62	719	760	669	669
query63	237	193	190	190
query64	1846	1147	828	828
query65	4469	4227	4240	4227
query66	779	406	312	312
query67	15776	15557	15372	15372
query68	6845	888	512	512
query69	539	294	264	264
query70	1183	1115	1075	1075
query71	491	323	288	288
query72	6056	4743	4782	4743
query73	1495	628	346	346
query74	8956	9224	9056	9056
query75	4059	3201	2697	2697
query76	4207	1191	760	760
query77	748	374	272	272
query78	10050	10098	9352	9352
query79	2621	801	558	558
query80	584	493	437	437
query81	490	251	226	226
query82	474	121	97	97
query83	293	254	225	225
query84	291	111	83	83
query85	776	349	313	313
query86	363	306	286	286
query87	4375	4352	4390	4352
query88	3720	2220	2276	2220
query89	407	326	288	288
query90	1829	218	216	216
query91	141	142	111	111
query92	69	63	54	54
query93	1999	931	580	580
query94	658	425	294	294
query95	370	306	287	287
query96	482	553	267	267
query97	3157	3260	3133	3133
query98	224	214	202	202
query99	1483	1467	1299	1299
Total cold run time: 297673 ms
Total hot run time: 192562 ms

@doris-robot
Copy link

doris-robot commented Apr 18, 2025

ClickBench: Total hot run time: 29.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1e2a77a25c8f52ce89ce0df3c667801f20bb3421, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.24	0.20	0.19
query4	1.59	0.19	0.19
query5	0.60	0.59	0.59
query6	1.19	0.72	0.73
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.49
query10	0.58	0.57	0.55
query11	0.16	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	1.13	1.17	1.18
query15	0.88	0.84	0.85
query16	0.39	0.37	0.38
query17	1.06	1.01	1.07
query18	0.22	0.20	0.20
query19	1.92	1.78	1.79
query20	0.01	0.01	0.01
query21	15.40	0.93	0.54
query22	0.74	1.11	0.66
query23	15.01	1.38	0.66
query24	7.14	1.38	1.18
query25	0.49	0.19	0.07
query26	0.65	0.16	0.13
query27	0.05	0.05	0.04
query28	8.92	0.88	0.41
query29	12.59	4.06	3.39
query30	0.26	0.09	0.06
query31	2.82	0.61	0.39
query32	3.22	0.54	0.48
query33	3.02	3.02	3.02
query34	15.77	5.15	4.48
query35	4.49	4.49	4.49
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.02
query40	0.18	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.29 s
Total hot run time: 29.89 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 18, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/29) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.15% (14422/27132)
Line Coverage 42.01% (124969/297472)
Region Coverage 40.83% (63847/156381)
Branch Coverage 35.46% (32109/90538)

@felixwluo felixwluo mentioned this pull request Apr 18, 2025
Merged
8 tasks
HappenLee
HappenLee reviewed Apr 18, 2025
View reviewed changes
be/src/vec/functions/function_string.h Outdated
while (num <= part_number && offset >= 0) {
offset = (int)substr.rfind(delimiter, offset);

std::string delimiter_str(reinterpret_cast<const char*>(delimiter.data),
Copy link
Contributor

@HappenLee HappenLee Apr 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use StringRef of std::string_view

@felixwluo felixwluo force-pushed the fix-substring-0417 branch from 1e2a77a to 0027702 Compare April 18, 2025 15:32
@felixwluo
Copy link
Contributor Author

felixwluo commented Apr 18, 2025

run buildall

@felixwluo felixwluo requested a review from HappenLee April 18, 2025 15:33
@doris-robot
Copy link

doris-robot commented Apr 18, 2025

TPC-H: Total hot run time: 34197 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 002770294cc6f91116fab047322369969c271d18, data reload: false

------ Round 1 ----------------------------------
q1	26134	5094	5058	5058
q2	2067	269	179	179
q3	10427	1242	698	698
q4	10226	1010	512	512
q5	7543	2354	2348	2348
q6	179	160	134	134
q7	942	735	612	612
q8	9303	1306	1030	1030
q9	6923	5109	5092	5092
q10	6909	2332	1954	1954
q11	501	284	278	278
q12	357	355	226	226
q13	17998	3708	3146	3146
q14	222	220	207	207
q15	526	480	495	480
q16	440	448	410	410
q17	601	856	367	367
q18	7531	7276	7209	7209
q19	1519	950	556	556
q20	339	325	222	222
q21	4023	3395	2492	2492
q22	1035	1006	987	987
Total cold run time: 115745 ms
Total hot run time: 34197 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5123	5087	5120	5087
q2	244	325	235	235
q3	2130	2597	2284	2284
q4	1449	1794	1403	1403
q5	4489	4409	4370	4370
q6	211	168	132	132
q7	2001	1905	1796	1796
q8	2602	2583	2548	2548
q9	7194	7253	7103	7103
q10	3010	3202	2718	2718
q11	564	489	474	474
q12	705	778	623	623
q13	3634	4040	3360	3360
q14	311	323	280	280
q15	518	476	484	476
q16	463	515	455	455
q17	1158	1535	1391	1391
q18	7639	7575	7596	7575
q19	814	838	876	838
q20	1977	1964	1849	1849
q21	5311	4966	4813	4813
q22	1115	1081	1017	1017
Total cold run time: 52662 ms
Total hot run time: 50827 ms

@doris-robot
Copy link

doris-robot commented Apr 18, 2025

TPC-DS: Total hot run time: 191897 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 002770294cc6f91116fab047322369969c271d18, data reload: false

query1	1413	1079	1059	1059
query2	6329	1793	1774	1774
query3	11060	4589	4492	4492
query4	53572	25313	23323	23323
query5	5338	523	452	452
query6	409	194	191	191
query7	5250	490	276	276
query8	322	236	211	211
query9	6940	2504	2517	2504
query10	410	321	251	251
query11	15332	14960	14947	14947
query12	156	110	102	102
query13	1210	500	400	400
query14	10005	6313	6164	6164
query15	196	196	173	173
query16	7000	659	529	529
query17	1096	706	565	565
query18	1542	406	307	307
query19	203	193	162	162
query20	129	123	113	113
query21	201	133	105	105
query22	4364	4441	4236	4236
query23	34103	33367	33396	33367
query24	6666	2401	2412	2401
query25	469	466	393	393
query26	632	287	147	147
query27	2160	507	345	345
query28	3148	2134	2126	2126
query29	547	557	435	435
query30	270	226	190	190
query31	868	862	779	779
query32	74	69	65	65
query33	442	381	340	340
query34	773	873	532	532
query35	819	863	774	774
query36	967	1001	893	893
query37	112	101	80	80
query38	4248	4152	4144	4144
query39	1478	1447	1407	1407
query40	215	116	117	116
query41	52	58	51	51
query42	118	105	105	105
query43	462	498	476	476
query44	1298	812	799	799
query45	190	179	170	170
query46	841	1024	641	641
query47	1823	1885	1814	1814
query48	374	404	308	308
query49	689	519	429	429
query50	651	741	427	427
query51	4294	4211	4193	4193
query52	115	109	100	100
query53	230	265	185	185
query54	561	555	500	500
query55	80	83	84	83
query56	309	307	325	307
query57	1203	1207	1109	1109
query58	268	258	245	245
query59	2688	2762	2671	2671
query60	331	321	311	311
query61	127	140	145	140
query62	747	756	682	682
query63	227	183	227	183
query64	1412	1014	681	681
query65	4419	4304	4314	4304
query66	721	417	312	312
query67	15876	15465	15248	15248
query68	4683	899	528	528
query69	514	303	264	264
query70	1206	1089	1097	1089
query71	440	316	290	290
query72	5772	4994	5066	4994
query73	852	693	347	347
query74	8899	9244	8630	8630
query75	3384	3200	2718	2718
query76	3356	1190	728	728
query77	539	378	280	280
query78	9957	10113	9315	9315
query79	2514	798	550	550
query80	637	531	425	425
query81	477	254	215	215
query82	615	128	96	96
query83	242	250	224	224
query84	304	101	85	85
query85	783	349	306	306
query86	464	303	286	286
query87	4486	4311	4353	4311
query88	3575	2192	2206	2192
query89	410	314	282	282
query90	1863	204	208	204
query91	140	140	111	111
query92	72	61	57	57
query93	2620	941	577	577
query94	688	418	287	287
query95	370	300	281	281
query96	485	555	268	268
query97	3164	3165	3114	3114
query98	230	216	209	209
query99	1367	1386	1290	1290
Total cold run time: 295682 ms
Total hot run time: 191897 ms

@doris-robot
Copy link

doris-robot commented Apr 18, 2025

ClickBench: Total hot run time: 29.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 002770294cc6f91116fab047322369969c271d18, data reload: false

query1	0.04	0.03	0.04
query2	0.14	0.11	0.11
query3	0.25	0.19	0.20
query4	1.59	0.19	0.19
query5	0.58	0.57	0.59
query6	1.17	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.53	0.53
query10	0.58	0.57	0.56
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.60	0.60	0.59
query14	1.15	1.16	1.21
query15	0.87	0.86	0.89
query16	0.38	0.37	0.39
query17	1.00	1.03	1.01
query18	0.21	0.20	0.20
query19	1.96	1.80	1.83
query20	0.01	0.01	0.01
query21	15.42	0.90	0.55
query22	0.74	1.27	0.64
query23	14.88	1.32	0.62
query24	6.50	1.36	0.93
query25	0.51	0.24	0.27
query26	0.54	0.16	0.13
query27	0.06	0.06	0.05
query28	10.00	0.83	0.44
query29	12.55	3.93	3.30
query30	0.26	0.09	0.07
query31	2.81	0.60	0.38
query32	3.22	0.55	0.46
query33	3.13	3.03	3.08
query34	15.79	5.08	4.56
query35	4.58	4.51	4.53
query36	0.67	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.62 s
Total hot run time: 29.85 s

@doris-robot
Copy link

doris-robot commented Apr 18, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/29) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.16% (14426/27137)
Line Coverage 42.01% (124957/297472)
Region Coverage 40.83% (63855/156374)
Branch Coverage 35.46% (32100/90530)

@felixwluo
Copy link
Contributor Author

felixwluo commented Apr 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 20, 2025

TPC-H: Total hot run time: 33561 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8324f3220477832989852d9c3e9cb043c72aa585, data reload: false

------ Round 1 ----------------------------------
q1	25885	5012	5018	5012
q2	2073	278	174	174
q3	10406	1231	666	666
q4	10211	1011	505	505
q5	7505	2270	2320	2270
q6	183	163	131	131
q7	901	779	617	617
q8	9305	1295	1048	1048
q9	6889	5101	5151	5101
q10	6799	2302	1880	1880
q11	462	277	264	264
q12	351	353	220	220
q13	17798	3713	3028	3028
q14	243	231	205	205
q15	523	478	494	478
q16	442	450	402	402
q17	616	852	363	363
q18	7408	7096	7014	7014
q19	1220	960	559	559
q20	337	332	221	221
q21	3816	2588	2416	2416
q22	1065	1024	987	987
Total cold run time: 114438 ms
Total hot run time: 33561 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5096	5115	5011	5011
q2	245	323	234	234
q3	2170	2627	2269	2269
q4	1393	1804	1358	1358
q5	4315	4326	4383	4326
q6	214	172	127	127
q7	1971	1952	1844	1844
q8	2592	2621	2522	2522
q9	7348	7369	7198	7198
q10	3032	3193	2744	2744
q11	589	503	491	491
q12	646	757	628	628
q13	3462	3893	3258	3258
q14	286	292	275	275
q15	532	490	487	487
q16	487	490	472	472
q17	1129	1568	1425	1425
q18	7871	7583	7460	7460
q19	765	797	798	797
q20	1990	1962	1820	1820
q21	5264	4926	4762	4762
q22	1169	1107	1090	1090
Total cold run time: 52566 ms
Total hot run time: 50598 ms

@doris-robot
Copy link

doris-robot commented Apr 20, 2025

TPC-DS: Total hot run time: 192315 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8324f3220477832989852d9c3e9cb043c72aa585, data reload: false

query1	1405	1096	1028	1028
query2	6449	1792	1764	1764
query3	11155	4538	4587	4538
query4	52664	25564	23413	23413
query5	5100	546	446	446
query6	351	219	195	195
query7	4886	501	288	288
query8	290	225	226	225
query9	5330	2525	2479	2479
query10	430	312	250	250
query11	15053	14945	15354	14945
query12	157	107	109	107
query13	1024	508	375	375
query14	10058	6266	6298	6266
query15	200	206	165	165
query16	7080	667	541	541
query17	1137	742	584	584
query18	1574	422	301	301
query19	198	186	171	171
query20	134	133	134	133
query21	209	127	110	110
query22	4384	4527	4330	4330
query23	34291	33266	33508	33266
query24	6590	2397	2421	2397
query25	456	466	400	400
query26	724	273	149	149
query27	2437	499	333	333
query28	3086	2126	2122	2122
query29	570	554	423	423
query30	270	221	211	211
query31	847	850	787	787
query32	75	60	65	60
query33	457	383	327	327
query34	768	896	502	502
query35	818	873	753	753
query36	937	996	891	891
query37	110	97	72	72
query38	4176	4190	4177	4177
query39	1511	1415	1433	1415
query40	213	121	105	105
query41	57	49	51	49
query42	124	112	103	103
query43	493	494	477	477
query44	1314	821	831	821
query45	180	176	166	166
query46	836	1004	638	638
query47	1832	1904	1816	1816
query48	382	431	318	318
query49	670	507	449	449
query50	661	693	418	418
query51	4216	4261	4254	4254
query52	109	101	94	94
query53	226	254	184	184
query54	556	573	504	504
query55	80	79	80	79
query56	369	302	298	298
query57	1189	1189	1139	1139
query58	280	257	255	255
query59	2678	2639	2636	2636
query60	337	326	302	302
query61	129	123	123	123
query62	747	733	628	628
query63	233	189	184	184
query64	1831	1031	769	769
query65	4453	4429	4251	4251
query66	739	402	304	304
query67	15907	15516	15251	15251
query68	8297	877	508	508
query69	530	297	277	277
query70	1210	1108	1051	1051
query71	493	328	289	289
query72	5710	4846	4818	4818
query73	1516	691	346	346
query74	8931	9078	8908	8908
query75	4199	3293	2735	2735
query76	4296	1186	749	749
query77	642	357	277	277
query78	9990	10125	9241	9241
query79	2522	822	574	574
query80	678	498	437	437
query81	481	251	216	216
query82	347	124	95	95
query83	341	243	225	225
query84	283	105	84	84
query85	766	410	309	309
query86	356	293	269	269
query87	4378	4443	4347	4347
query88	2829	2216	2242	2216
query89	395	315	278	278
query90	2109	209	215	209
query91	141	141	111	111
query92	77	58	56	56
query93	1727	966	576	576
query94	679	414	312	312
query95	366	300	287	287
query96	481	571	273	273
query97	3169	3223	3213	3213
query98	221	208	203	203
query99	1405	1409	1272	1272
Total cold run time: 297797 ms
Total hot run time: 192315 ms

@doris-robot
Copy link

doris-robot commented Apr 20, 2025

ClickBench: Total hot run time: 29.97 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8324f3220477832989852d9c3e9cb043c72aa585, data reload: false

query1	0.03	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.19	0.20
query4	1.59	0.19	0.18
query5	0.59	0.58	0.59
query6	1.19	0.71	0.73
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.51	0.51
query10	0.56	0.57	0.56
query11	0.16	0.12	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	1.17	1.22	1.20
query15	0.86	0.84	0.84
query16	0.38	0.39	0.38
query17	1.01	1.06	1.01
query18	0.22	0.20	0.20
query19	1.90	1.75	1.83
query20	0.01	0.01	0.02
query21	15.39	0.91	0.54
query22	0.76	1.19	0.76
query23	14.84	1.39	0.66
query24	7.10	1.65	1.09
query25	0.52	0.11	0.12
query26	0.60	0.16	0.14
query27	0.06	0.06	0.05
query28	10.58	0.81	0.45
query29	12.89	3.99	3.31
query30	0.26	0.09	0.06
query31	2.82	0.58	0.38
query32	3.24	0.55	0.47
query33	2.99	3.08	3.04
query34	15.82	5.13	4.49
query35	4.54	4.53	4.50
query36	0.66	0.49	0.48
query37	0.08	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.99 s
Total hot run time: 29.97 s

@doris-robot
Copy link

doris-robot commented Apr 20, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/29) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.16% (14427/27139)
Line Coverage 42.02% (125020/297511)
Region Coverage 40.84% (63880/156403)
Branch Coverage 35.47% (32113/90544)

@doris-robot
Copy link

doris-robot commented Apr 21, 2025

TPC-DS: Total hot run time: 184369 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ef1a511e0c4ce2b1e7262357a445a9bcc077916, data reload: false

query1	1055	475	505	475
query2	6565	1815	1781	1781
query3	6741	220	217	217
query4	26160	23639	23064	23064
query5	4286	625	455	455
query6	281	188	176	176
query7	4623	478	267	267
query8	293	228	232	228
query9	8614	2553	2565	2553
query10	486	315	269	269
query11	15451	15097	14853	14853
query12	165	111	106	106
query13	1659	542	382	382
query14	9421	5976	6163	5976
query15	207	190	162	162
query16	7306	662	465	465
query17	1163	684	556	556
query18	1961	396	302	302
query19	189	174	153	153
query20	112	116	118	116
query21	206	129	106	106
query22	4026	4104	4005	4005
query23	33775	32869	33009	32869
query24	8426	2425	2374	2374
query25	537	453	378	378
query26	1231	268	148	148
query27	2748	499	325	325
query28	4365	2099	2073	2073
query29	777	540	409	409
query30	280	213	185	185
query31	939	844	751	751
query32	80	64	62	62
query33	554	354	328	328
query34	786	858	524	524
query35	785	804	727	727
query36	941	980	886	886
query37	121	100	81	81
query38	4264	4188	4081	4081
query39	1446	1392	1426	1392
query40	217	126	110	110
query41	63	62	62	62
query42	119	109	106	106
query43	481	493	471	471
query44	1273	789	792	789
query45	175	176	170	170
query46	838	1017	616	616
query47	1749	1774	1730	1730
query48	374	406	306	306
query49	797	569	397	397
query50	643	672	403	403
query51	4092	4071	4126	4071
query52	104	106	95	95
query53	217	247	188	188
query54	571	565	490	490
query55	86	80	84	80
query56	309	290	280	280
query57	1164	1134	1061	1061
query58	265	256	248	248
query59	2619	2658	2559	2559
query60	326	314	308	308
query61	162	124	125	124
query62	795	725	663	663
query63	226	179	190	179
query64	4253	985	663	663
query65	4369	4223	4236	4223
query66	1133	421	308	308
query67	15621	15361	15229	15229
query68	7804	864	512	512
query69	459	308	257	257
query70	1187	1153	1110	1110
query71	412	313	281	281
query72	5561	4700	4626	4626
query73	645	586	339	339
query74	8856	9122	8612	8612
query75	3388	3187	2673	2673
query76	3347	1186	751	751
query77	547	376	286	286
query78	9929	10001	9264	9264
query79	1823	876	561	561
query80	665	588	447	447
query81	493	253	222	222
query82	404	124	98	98
query83	254	252	232	232
query84	256	113	80	80
query85	751	363	300	300
query86	390	300	287	287
query87	4441	4335	4343	4335
query88	3035	2209	2222	2209
query89	378	313	282	282
query90	1936	211	213	211
query91	155	145	112	112
query92	81	60	55	55
query93	2466	952	584	584
query94	668	402	286	286
query95	376	287	283	283
query96	487	557	280	280
query97	3072	3236	3114	3114
query98	219	206	204	204
query99	1320	1403	1287	1287
Total cold run time: 271871 ms
Total hot run time: 184369 ms

@doris-robot
Copy link

doris-robot commented Apr 21, 2025

ClickBench: Total hot run time: 29.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3ef1a511e0c4ce2b1e7262357a445a9bcc077916, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.26	0.18	0.19
query4	1.59	0.19	0.19
query5	0.58	0.59	0.59
query6	1.17	0.72	0.73
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.53	0.50
query10	0.58	0.57	0.56
query11	0.15	0.11	0.10
query12	0.15	0.12	0.11
query13	0.61	0.60	0.60
query14	1.15	1.22	1.16
query15	0.86	0.85	0.85
query16	0.38	0.38	0.39
query17	1.00	1.04	1.01
query18	0.20	0.19	0.19
query19	1.86	1.84	1.80
query20	0.01	0.01	0.02
query21	15.41	0.93	0.55
query22	0.77	1.20	0.63
query23	14.95	1.36	0.61
query24	7.20	1.08	0.36
query25	0.47	0.20	0.12
query26	0.62	0.17	0.14
query27	0.05	0.05	0.04
query28	9.06	0.93	0.45
query29	12.51	4.05	3.37
query30	0.24	0.08	0.06
query31	2.82	0.59	0.40
query32	3.23	0.55	0.46
query33	3.02	3.01	3.06
query34	15.80	5.02	4.49
query35	4.50	4.49	4.49
query36	0.67	0.49	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.18 s
Total hot run time: 29.12 s

@felixwluo
Copy link
Contributor Author

felixwluo commented Apr 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 21, 2025

TPC-H: Total hot run time: 33949 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 381a131f23c7c70b9b09caaa3edc8ae9e9056274, data reload: false

------ Round 1 ----------------------------------
q1	26240	5727	5045	5045
q2	2091	278	192	192
q3	10388	1217	705	705
q4	10248	977	552	552
q5	7551	2366	2288	2288
q6	181	159	133	133
q7	910	727	599	599
q8	9299	1264	1083	1083
q9	6947	5124	5083	5083
q10	7197	2305	1885	1885
q11	484	275	271	271
q12	356	350	224	224
q13	17817	3658	3170	3170
q14	228	232	208	208
q15	528	478	489	478
q16	460	447	412	412
q17	596	837	367	367
q18	7584	7176	7033	7033
q19	1225	956	554	554
q20	313	329	217	217
q21	4319	3353	2491	2491
q22	1033	982	959	959
Total cold run time: 115995 ms
Total hot run time: 33949 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5132	5110	5073	5073
q2	232	335	239	239
q3	2209	2649	2278	2278
q4	1444	1820	1496	1496
q5	4477	4335	4360	4335
q6	217	168	132	132
q7	1968	1924	1758	1758
q8	2595	2591	2551	2551
q9	7294	7011	7078	7011
q10	3018	3196	2738	2738
q11	584	514	500	500
q12	703	745	601	601
q13	3489	3931	3460	3460
q14	285	298	268	268
q15	526	503	495	495
q16	491	506	455	455
q17	1157	1571	1401	1401
q18	7894	7523	7559	7523
q19	802	817	884	817
q20	1921	1938	1889	1889
q21	5271	4949	4808	4808
q22	1092	1069	1012	1012
Total cold run time: 52801 ms
Total hot run time: 50840 ms

@doris-robot
Copy link

doris-robot commented Apr 21, 2025

TPC-DS: Total hot run time: 192729 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 381a131f23c7c70b9b09caaa3edc8ae9e9056274, data reload: false

query1	1410	1090	1071	1071
query2	6050	1800	1815	1800
query3	10997	4640	4667	4640
query4	52893	25690	22938	22938
query5	5170	583	463	463
query6	354	187	192	187
query7	4867	484	274	274
query8	306	247	233	233
query9	5425	2552	2557	2552
query10	413	298	254	254
query11	14985	15022	14889	14889
query12	162	110	101	101
query13	1023	524	397	397
query14	10178	6547	6606	6547
query15	217	211	168	168
query16	7092	655	548	548
query17	1148	722	630	630
query18	1589	425	336	336
query19	211	212	179	179
query20	138	136	129	129
query21	215	134	106	106
query22	4449	4377	4384	4377
query23	33985	33487	33475	33475
query24	7065	2478	2453	2453
query25	469	478	404	404
query26	731	273	152	152
query27	2411	526	344	344
query28	2918	2150	2145	2145
query29	622	583	438	438
query30	273	224	188	188
query31	856	850	778	778
query32	71	82	62	62
query33	471	373	317	317
query34	790	881	516	516
query35	813	828	796	796
query36	972	1029	894	894
query37	115	107	76	76
query38	4317	4183	4292	4183
query39	1488	1435	1484	1435
query40	237	120	113	113
query41	54	54	52	52
query42	124	113	116	113
query43	509	525	481	481
query44	1377	819	813	813
query45	186	178	166	166
query46	859	1036	668	668
query47	1854	1893	1758	1758
query48	378	417	311	311
query49	693	507	425	425
query50	661	712	415	415
query51	4211	4242	4125	4125
query52	112	113	106	106
query53	235	262	190	190
query54	599	593	506	506
query55	81	84	83	83
query56	305	307	282	282
query57	1167	1195	1109	1109
query58	266	259	280	259
query59	2667	2688	2624	2624
query60	344	336	299	299
query61	129	142	132	132
query62	744	752	668	668
query63	226	202	197	197
query64	2096	1111	709	709
query65	4403	4325	4319	4319
query66	777	443	309	309
query67	15957	15403	15281	15281
query68	7373	898	522	522
query69	538	317	270	270
query70	1211	1100	1116	1100
query71	501	307	307	307
query72	5862	4783	4962	4783
query73	1477	713	346	346
query74	9174	9136	8772	8772
query75	3988	3182	2786	2786
query76	4311	1244	780	780
query77	661	390	297	297
query78	9896	10087	9259	9259
query79	3316	846	570	570
query80	665	524	480	480
query81	484	256	223	223
query82	452	126	96	96
query83	331	254	242	242
query84	303	117	91	91
query85	787	363	313	313
query86	339	294	294	294
query87	4324	4483	4286	4286
query88	2974	2178	2199	2178
query89	438	317	288	288
query90	2002	222	229	222
query91	147	144	113	113
query92	74	65	57	57
query93	2048	955	584	584
query94	679	426	305	305
query95	368	302	290	290
query96	497	602	272	272
query97	3213	3218	3116	3116
query98	237	209	202	202
query99	1427	1398	1297	1297
Total cold run time: 299007 ms
Total hot run time: 192729 ms

@doris-robot
Copy link

doris-robot commented Apr 21, 2025

ClickBench: Total hot run time: 29.47 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 381a131f23c7c70b9b09caaa3edc8ae9e9056274, data reload: false

query1	0.04	0.04	0.03
query2	0.11	0.11	0.11
query3	0.25	0.19	0.19
query4	1.60	0.18	0.19
query5	0.59	0.58	0.59
query6	1.18	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.58	0.53	0.52
query10	0.57	0.57	0.56
query11	0.15	0.11	0.10
query12	0.15	0.11	0.11
query13	0.62	0.60	0.60
query14	1.19	1.16	1.18
query15	0.86	0.86	0.85
query16	0.39	0.38	0.38
query17	1.01	1.05	1.01
query18	0.21	0.20	0.20
query19	1.95	1.86	1.82
query20	0.02	0.01	0.01
query21	15.41	0.94	0.55
query22	0.77	1.30	0.77
query23	14.74	1.37	0.64
query24	7.83	0.93	0.50
query25	0.52	0.12	0.13
query26	0.71	0.17	0.12
query27	0.05	0.04	0.05
query28	8.99	0.88	0.42
query29	12.56	4.06	3.40
query30	0.27	0.09	0.06
query31	2.82	0.59	0.38
query32	3.22	0.55	0.46
query33	3.00	3.10	3.08
query34	16.16	5.10	4.46
query35	4.51	4.49	4.50
query36	0.65	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.17	0.15	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.24 s
Total hot run time: 29.47 s

@felixwluo
Copy link
Contributor Author

felixwluo commented Apr 22, 2025

run p0

@felixwluo
Copy link
Contributor Author

felixwluo commented Apr 22, 2025

run cloud_p0

@felixwluo felixwluo requested a review from yiguolei April 22, 2025 07:19
zhiqiang-hhhh
zhiqiang-hhhh approved these changes Apr 22, 2025
View reviewed changes
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 22, 2025
yiguolei
yiguolei approved these changes Apr 22, 2025
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 22, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 22, 2025

PR approved by at least one committer and no changes requested.

@yiguolei yiguolei merged commit 1f03d4b into apache:master Apr 22, 2025
24 of 26 checks passed
github-actions bot pushed a commit that referenced this pull request Apr 22, 2025
@felixwluo
[feat](function) SUBSTRING_INDEX function delimiter supports dynamic (#…
13ee911 
…50149)

### What problem does this PR solve?

Problem Summary:

sql
```
SELECT t1.no
      ,t1.sub_str
      ,t1.str
      ,substring_index(t1.str, t1.sub_str, -1)
      ,t2.rst2
FROM (
       SELECT 1 AS no, 'BBB' AS sub_str, 'AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 2 AS no, 'ccc' AS sub_str, 'zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06' AS str UNION ALL
       SELECT 3 AS no, 'DDD' AS sub_str, 'AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 4 AS no, 'DDD' AS sub_str, 'sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06' AS str UNION ALL
       SELECT 5 AS no, 'eee' AS sub_str, 'cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 6 AS no, 'A_01' AS sub_str, 'AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06' AS str 
     ) t1
LEFT JOIN (
       SELECT 1 AS no, 'BBB' AS sub_str,  substring_index('AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06', 'BBB', -1) AS rst2 UNION ALL
       SELECT 2 AS no, 'ccc' AS sub_str,  substring_index('zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06', 'ccc', -1) AS rst2 UNION ALL
       SELECT 3 AS no, 'DDD' AS sub_str,  substring_index('AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06', 'DDD', -1) AS rst2 UNION ALL
       SELECT 4 AS no, 'DDD' AS sub_str,  substring_index('sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06', 'DDD', -1) AS rst2 UNION ALL
       SELECT 5 AS no, 'eee' AS sub_str,  substring_index('cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06', 'eee', -1) AS rst2 UNION ALL
       SELECT 6 AS no, 'A_01' AS sub_str, substring_index('AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06', 'A_01', -1) AS rst2 
     ) t2
     ON t1.no = t2.no AND t1.sub_str = t2.sub_str
ORDER BY t1.no;
```
previous results for doris:
```
-- master
function except for the first argument, other parameter must be a constant

-- 2.1
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+
| no   | sub_str | str                                       | substring_index(`t1`.`str`, `t1`.`sub_str`, -1) | rst2                                      |
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+
|    1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06                 | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|    2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06       | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|    3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06                 | _04|EEE_05|FFF_06                         |
|    4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06       | _04|rfv_05|rgb_06                         |
|    5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06       | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|    6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+
```

mysql result:
```
+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| no | sub_str | str                                       | substring_index(t1.str, t1.sub_str, -1)   | rst2                                      |
+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|  1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06           | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|  2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|  3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _04|EEE_05|FFF_06                         | _04|EEE_05|FFF_06                         |
|  4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | _04|rfv_05|rgb_06                         | _04|rfv_05|rgb_06                         |
|  5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|  6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
```

current doris results:
```
+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| no   | sub_str | str                                       | substring_index(t1.str, t1.sub_str, -1)   | rst2                                      |
+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|    1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06           | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|    2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|    3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _04|EEE_05|FFF_06                         | _04|EEE_05|FFF_06                         |
|    4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | _04|rfv_05|rgb_06                         | _04|rfv_05|rgb_06                         |
|    5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|    6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
```

now consistent with mysql behavior.
@github-actions github-actions bot mentioned this pull request Apr 22, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Apr 22, 2025
@felixwluo
[feat](function) SUBSTRING_INDEX function delimiter supports dynamic (#…
d658584 
…50149)

### What problem does this PR solve?

Problem Summary:

sql
```
SELECT t1.no
      ,t1.sub_str
      ,t1.str
      ,substring_index(t1.str, t1.sub_str, -1)
      ,t2.rst2
FROM (
       SELECT 1 AS no, 'BBB' AS sub_str, 'AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 2 AS no, 'ccc' AS sub_str, 'zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06' AS str UNION ALL
       SELECT 3 AS no, 'DDD' AS sub_str, 'AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 4 AS no, 'DDD' AS sub_str, 'sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06' AS str UNION ALL
       SELECT 5 AS no, 'eee' AS sub_str, 'cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 6 AS no, 'A_01' AS sub_str, 'AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06' AS str 
     ) t1
LEFT JOIN (
       SELECT 1 AS no, 'BBB' AS sub_str,  substring_index('AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06', 'BBB', -1) AS rst2 UNION ALL
       SELECT 2 AS no, 'ccc' AS sub_str,  substring_index('zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06', 'ccc', -1) AS rst2 UNION ALL
       SELECT 3 AS no, 'DDD' AS sub_str,  substring_index('AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06', 'DDD', -1) AS rst2 UNION ALL
       SELECT 4 AS no, 'DDD' AS sub_str,  substring_index('sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06', 'DDD', -1) AS rst2 UNION ALL
       SELECT 5 AS no, 'eee' AS sub_str,  substring_index('cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06', 'eee', -1) AS rst2 UNION ALL
       SELECT 6 AS no, 'A_01' AS sub_str, substring_index('AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06', 'A_01', -1) AS rst2 
     ) t2
     ON t1.no = t2.no AND t1.sub_str = t2.sub_str
ORDER BY t1.no;
```
previous results for doris:
```
-- master
function except for the first argument, other parameter must be a constant

-- 2.1
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+
| no   | sub_str | str                                       | substring_index(`t1`.`str`, `t1`.`sub_str`, -1) | rst2                                      |
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+
|    1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06                 | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|    2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06       | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|    3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06                 | _04|EEE_05|FFF_06                         |
|    4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06       | _04|rfv_05|rgb_06                         |
|    5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06       | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|    6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+
```

mysql result:
```
+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| no | sub_str | str                                       | substring_index(t1.str, t1.sub_str, -1)   | rst2                                      |
+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|  1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06           | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|  2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|  3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _04|EEE_05|FFF_06                         | _04|EEE_05|FFF_06                         |
|  4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | _04|rfv_05|rgb_06                         | _04|rfv_05|rgb_06                         |
|  5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|  6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
```

current doris results:
```
+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| no   | sub_str | str                                       | substring_index(t1.str, t1.sub_str, -1)   | rst2                                      |
+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|    1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06           | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|    2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|    3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _04|EEE_05|FFF_06                         | _04|EEE_05|FFF_06                         |
|    4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | _04|rfv_05|rgb_06                         | _04|rfv_05|rgb_06                         |
|    5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|    6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
```

now consistent with mysql behavior.
@github-actions github-actions bot mentioned this pull request Apr 22, 2025
Merged
yiguolei pushed a commit that referenced this pull request Apr 23, 2025
@github-actions @felixwluo
branch-2.1: [feat](function) SUBSTRING_INDEX function delimiter suppo…
98f5d95 
…rts dynamic #50149 (#50303)

Cherry-picked from #50149

Co-authored-by: lw112 <131352377+felixwluo@users.noreply.github.com>
dataroaring pushed a commit that referenced this pull request Apr 27, 2025
@github-actions @felixwluo
branch-3.0: [feat](function) SUBSTRING_INDEX function delimiter suppo…
f7153b7 
…rts dynamic #50149 (#50302)

Cherry-picked from #50149

Co-authored-by: lw112 <131352377+felixwluo@users.noreply.github.com>
@yiguolei yiguolei mentioned this pull request May 13, 2025
Closed
KassieZ pushed a commit to apache/doris-website that referenced this pull request May 20, 2025
@felixwluo
Remove the description that the parameter does not support variables (#…
14a009a 
…2300)

because: apache/doris#50149

## Versions 

- [x] dev
- [x] 3.0
- [x] 2.1
- [ ] 2.0

## Languages

- [x] Chinese
- [x] English

## Docs Checklist

- [ ] Checked by AI
- [ ] Test Cases Built
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@felixwluo @koarz
[feat](function) SUBSTRING_INDEX function delimiter supports dynamic (a…
6ad6663 
…pache#50149)

### What problem does this PR solve?

Problem Summary:

sql
```
SELECT t1.no
      ,t1.sub_str
      ,t1.str
      ,substring_index(t1.str, t1.sub_str, -1)
      ,t2.rst2
FROM (
       SELECT 1 AS no, 'BBB' AS sub_str, 'AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 2 AS no, 'ccc' AS sub_str, 'zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06' AS str UNION ALL
       SELECT 3 AS no, 'DDD' AS sub_str, 'AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 4 AS no, 'DDD' AS sub_str, 'sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06' AS str UNION ALL
       SELECT 5 AS no, 'eee' AS sub_str, 'cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06' AS str UNION ALL
       SELECT 6 AS no, 'A_01' AS sub_str, 'AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06' AS str 
     ) t1
LEFT JOIN (
       SELECT 1 AS no, 'BBB' AS sub_str,  substring_index('AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06', 'BBB', -1) AS rst2 UNION ALL
       SELECT 2 AS no, 'ccc' AS sub_str,  substring_index('zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06', 'ccc', -1) AS rst2 UNION ALL
       SELECT 3 AS no, 'DDD' AS sub_str,  substring_index('AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06', 'DDD', -1) AS rst2 UNION ALL
       SELECT 4 AS no, 'DDD' AS sub_str,  substring_index('sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06', 'DDD', -1) AS rst2 UNION ALL
       SELECT 5 AS no, 'eee' AS sub_str,  substring_index('cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06', 'eee', -1) AS rst2 UNION ALL
       SELECT 6 AS no, 'A_01' AS sub_str, substring_index('AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06', 'A_01', -1) AS rst2 
     ) t2
     ON t1.no = t2.no AND t1.sub_str = t2.sub_str
ORDER BY t1.no;
```
previous results for doris:
```
-- master
function except for the first argument, other parameter must be a constant

-- 2.1
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+
| no   | sub_str | str                                       | substring_index(`t1`.`str`, `t1`.`sub_str`, -1) | rst2                                      |
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+
|    1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06                 | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|    2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06       | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|    3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06                 | _04|EEE_05|FFF_06                         |
|    4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06       | _04|rfv_05|rgb_06                         |
|    5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06       | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|    6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+------+---------+-------------------------------------------+-------------------------------------------------+-------------------------------------------+
```

mysql result:
```
+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| no | sub_str | str                                       | substring_index(t1.str, t1.sub_str, -1)   | rst2                                      |
+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|  1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06           | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|  2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|  3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _04|EEE_05|FFF_06                         | _04|EEE_05|FFF_06                         |
|  4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | _04|rfv_05|rgb_06                         | _04|rfv_05|rgb_06                         |
|  5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|  6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+----+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
```

current doris results:
```
+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
| no   | sub_str | str                                       | substring_index(t1.str, t1.sub_str, -1)   | rst2                                      |
+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
|    1 | BBB     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _02|CCC_03|DDD_04|EEE_05|FFF_06           | _02|CCC_03|DDD_04|EEE_05|FFF_06           |
|    2 | ccc     | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 | zyz_01|zyz_02|CCC_03|qwe_04|qwe_05|qwe_06 |
|    3 | DDD     | AAA_01|BBB_02|CCC_03|DDD_04|EEE_05|FFF_06 | _04|EEE_05|FFF_06                         | _04|EEE_05|FFF_06                         |
|    4 | DDD     | sgr_01|wsc_02|CCC_03|DDD_04|rfv_05|rgb_06 | _04|rfv_05|rgb_06                         | _04|rfv_05|rgb_06                         |
|    5 | eee     | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 | cdr_01|vfr_02|dfc_03|DDD_04|EEE_05|FFF_06 |
|    6 | A_01    | AAA_01|dsd_02|ert_03|bgt_04|fgh_05|hyb_06 | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       | |dsd_02|ert_03|bgt_04|fgh_05|hyb_06       |
+------+---------+-------------------------------------------+-------------------------------------------+-------------------------------------------+
```

now consistent with mysql behavior.
@gavinchou gavinchou mentioned this pull request Jun 11, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@HappenLee HappenLee Awaiting requested review from HappenLee

+1 more reviewer

@zhiqiang-hhhh zhiqiang-hhhh zhiqiang-hhhh approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.6-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@felixwluo @Thearas @doris-robot @hello-stephen @yiguolei @HappenLee @zhiqiang-hhhh @dataroaring