Skip to content

Conversation

@LiBinfeng-01
Copy link
Contributor

@LiBinfeng-01 LiBinfeng-01 commented Mar 27, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

  • testFoldConst cannot throw exception to tell user whether the result is expected
  • codePointCount is used to count codePoints in explictly length, offsetByCodePoints is used to count length of codePoints
    example:
    "😊😉".codePointCount(0, 2) ==> 1
    "😊😉".offsetByCodePoints(0, 1) ==> 2
    because emoji would take two Surrogates to express. So we need to use offsetByCodePoints to express actually length we need
    in sql. While it would throw exception, so we need to consider boundary. So every function including offsetByCodePoints need to check boundary before calling it

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@LiBinfeng-01 LiBinfeng-01 force-pushed the fix_fold_constant_regression branch from 5ad1c80 to 2c38e6b Compare March 27, 2025 07:33
@LiBinfeng-01
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34461 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c38e6b970746fea7fe9a98eb341ae62d34f557c, data reload: false

------ Round 1 ----------------------------------
q1	25850	5168	5192	5168
q2	2077	280	161	161
q3	10412	1239	715	715
q4	10215	1007	536	536
q5	7508	2317	2390	2317
q6	205	165	133	133
q7	907	774	621	621
q8	9308	1298	1123	1123
q9	6834	5164	5132	5132
q10	6808	2323	1906	1906
q11	496	278	260	260
q12	357	370	220	220
q13	17758	3678	3080	3080
q14	235	236	220	220
q15	556	502	494	494
q16	622	624	592	592
q17	580	847	358	358
q18	7505	7214	7162	7162
q19	1549	961	590	590
q20	331	332	198	198
q21	3948	3270	2499	2499
q22	1026	1017	976	976
Total cold run time: 115087 ms
Total hot run time: 34461 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5215	5198	5170	5170
q2	233	322	234	234
q3	2153	2692	2305	2305
q4	1457	1838	1494	1494
q5	4571	4523	4466	4466
q6	216	178	130	130
q7	2074	1948	1840	1840
q8	2617	2623	2608	2608
q9	7429	7271	7117	7117
q10	3015	3194	2745	2745
q11	591	529	501	501
q12	682	750	629	629
q13	3523	3935	3287	3287
q14	295	319	278	278
q15	533	470	464	464
q16	637	690	655	655
q17	1175	1609	1358	1358
q18	7827	7688	7465	7465
q19	853	830	874	830
q20	1963	1987	1828	1828
q21	5380	4947	4827	4827
q22	1105	1124	985	985
Total cold run time: 53544 ms
Total hot run time: 51216 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193717 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c38e6b970746fea7fe9a98eb341ae62d34f557c, data reload: false

query1	1389	1079	1045	1045
query2	6144	1949	1883	1883
query3	11117	4677	4769	4677
query4	25770	23860	23139	23139
query5	4531	669	465	465
query6	307	194	195	194
query7	4000	503	281	281
query8	301	255	244	244
query9	8543	2596	2580	2580
query10	484	301	259	259
query11	15687	15313	14988	14988
query12	171	111	107	107
query13	1570	524	400	400
query14	9209	6253	6417	6253
query15	221	199	176	176
query16	7631	663	470	470
query17	1210	768	622	622
query18	2035	420	319	319
query19	208	203	191	191
query20	131	125	117	117
query21	213	123	113	113
query22	4474	4524	4294	4294
query23	34346	33617	33470	33470
query24	8563	2473	2460	2460
query25	508	476	405	405
query26	1252	275	144	144
query27	2731	519	333	333
query28	4628	2462	2450	2450
query29	755	577	434	434
query30	285	230	197	197
query31	925	867	782	782
query32	76	61	64	61
query33	550	365	309	309
query34	788	864	565	565
query35	828	851	776	776
query36	986	1018	921	921
query37	122	123	85	85
query38	4142	4250	4366	4250
query39	1487	1453	1425	1425
query40	215	120	106	106
query41	57	52	52	52
query42	134	112	110	110
query43	501	521	474	474
query44	1318	824	816	816
query45	181	175	167	167
query46	874	1044	663	663
query47	1853	1891	1760	1760
query48	386	429	304	304
query49	798	534	458	458
query50	702	773	439	439
query51	4347	4316	4257	4257
query52	105	105	105	105
query53	234	264	192	192
query54	524	511	422	422
query55	82	79	85	79
query56	273	280	284	280
query57	1202	1227	1139	1139
query58	249	254	240	240
query59	2731	2889	2706	2706
query60	287	309	279	279
query61	149	147	128	128
query62	779	744	694	694
query63	233	187	190	187
query64	3911	1087	692	692
query65	4443	4375	4442	4375
query66	1043	415	306	306
query67	16602	15564	15533	15533
query68	9160	838	518	518
query69	445	322	271	271
query70	1214	1090	1117	1090
query71	471	305	268	268
query72	5096	4700	4651	4651
query73	662	554	356	356
query74	9091	9230	9109	9109
query75	4107	3235	2723	2723
query76	3941	1199	763	763
query77	922	373	277	277
query78	10017	10340	9388	9388
query79	1848	802	610	610
query80	611	530	440	440
query81	483	255	226	226
query82	460	131	97	97
query83	178	176	161	161
query84	233	103	84	84
query85	760	367	318	318
query86	331	322	291	291
query87	4471	4606	4255	4255
query88	3502	2240	2252	2240
query89	391	309	279	279
query90	1914	206	211	206
query91	141	140	111	111
query92	74	60	56	56
query93	1297	1073	591	591
query94	662	414	314	314
query95	353	277	275	275
query96	501	551	280	280
query97	3216	3232	3124	3124
query98	226	203	202	202
query99	1444	1370	1296	1296
Total cold run time: 280753 ms
Total hot run time: 193717 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.53 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c38e6b970746fea7fe9a98eb341ae62d34f557c, data reload: false

query1	0.04	0.04	0.03
query2	0.11	0.10	0.10
query3	0.24	0.19	0.20
query4	1.59	0.19	0.18
query5	0.60	0.58	0.59
query6	1.21	0.71	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.58	0.51	0.50
query10	0.59	0.56	0.56
query11	0.16	0.10	0.11
query12	0.15	0.12	0.11
query13	0.62	0.60	0.61
query14	2.65	2.71	2.79
query15	0.93	0.86	0.85
query16	0.38	0.37	0.39
query17	1.01	1.06	1.01
query18	0.22	0.20	0.20
query19	1.93	1.91	1.89
query20	0.01	0.01	0.01
query21	15.36	0.95	0.56
query22	0.76	1.28	0.76
query23	14.75	1.35	0.65
query24	7.20	1.44	0.92
query25	0.53	0.16	0.12
query26	0.66	0.16	0.14
query27	0.05	0.05	0.05
query28	9.62	0.89	0.43
query29	12.53	4.01	3.32
query30	0.26	0.10	0.07
query31	2.84	0.58	0.38
query32	3.22	0.54	0.48
query33	3.03	3.05	3.04
query34	15.69	5.21	4.54
query35	4.59	4.53	4.56
query36	0.68	0.50	0.48
query37	0.09	0.06	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 105.34 s
Total hot run time: 31.53 s

morrySnow
morrySnow previously approved these changes Mar 27, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 27, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@LiBinfeng-01
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 28, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 34340 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b9ecb1be58c4a6dfecd2cb577548619b194196b3, data reload: false

------ Round 1 ----------------------------------
q1	25634	5045	5060	5045
q2	2067	292	166	166
q3	10489	1262	664	664
q4	10510	1009	532	532
q5	9118	2396	2285	2285
q6	184	164	133	133
q7	915	744	613	613
q8	9315	1272	1065	1065
q9	7925	5170	5143	5143
q10	6801	2296	1919	1919
q11	458	266	255	255
q12	354	352	217	217
q13	17933	3926	3483	3483
q14	254	250	226	226
q15	540	494	495	494
q16	614	623	598	598
q17	875	852	356	356
q18	10056	7638	7075	7075
q19	1797	953	559	559
q20	326	320	189	189
q21	4058	2614	2354	2354
q22	1010	1017	969	969
Total cold run time: 121233 ms
Total hot run time: 34340 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5140	5084	5068	5068
q2	230	326	234	234
q3	2114	2625	2381	2381
q4	1479	1884	1528	1528
q5	4390	4325	4325	4325
q6	212	166	131	131
q7	1947	1934	1780	1780
q8	2559	2574	2557	2557
q9	7201	7142	7154	7142
q10	2934	3160	2710	2710
q11	567	501	482	482
q12	677	745	596	596
q13	3456	3910	3284	3284
q14	298	308	272	272
q15	515	482	472	472
q16	656	696	660	660
q17	1134	1395	1369	1369
q18	7719	7656	7468	7468
q19	801	802	857	802
q20	1992	2052	1955	1955
q21	5461	4779	4778	4778
q22	1103	1054	1044	1044
Total cold run time: 52585 ms
Total hot run time: 51038 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192044 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b9ecb1be58c4a6dfecd2cb577548619b194196b3, data reload: false

query1	1370	1050	1027	1027
query2	6289	1903	1871	1871
query3	11146	4541	4521	4521
query4	52934	24022	22970	22970
query5	5032	588	474	474
query6	346	207	182	182
query7	4933	510	275	275
query8	339	251	237	237
query9	5929	2526	2530	2526
query10	410	320	261	261
query11	15140	14987	14919	14919
query12	165	108	106	106
query13	1104	534	412	412
query14	10020	6287	6521	6287
query15	200	191	174	174
query16	7050	678	494	494
query17	1079	736	613	613
query18	1538	427	336	336
query19	197	192	168	168
query20	131	130	128	128
query21	223	128	116	116
query22	4615	4655	4342	4342
query23	34022	33328	33349	33328
query24	6568	2426	2397	2397
query25	485	477	401	401
query26	715	275	142	142
query27	2282	508	329	329
query28	3251	2436	2413	2413
query29	613	557	438	438
query30	273	224	192	192
query31	865	882	792	792
query32	73	65	70	65
query33	475	358	299	299
query34	750	868	503	503
query35	799	832	766	766
query36	954	1036	935	935
query37	151	96	77	77
query38	4067	4123	4240	4123
query39	1494	1445	1440	1440
query40	204	115	101	101
query41	54	50	50	50
query42	124	110	103	103
query43	491	501	474	474
query44	1305	802	814	802
query45	173	172	174	172
query46	852	1024	630	630
query47	1876	1918	1790	1790
query48	394	422	303	303
query49	678	504	433	433
query50	709	733	422	422
query51	4321	4245	4242	4242
query52	105	104	100	100
query53	223	271	178	178
query54	494	494	428	428
query55	83	84	81	81
query56	293	273	265	265
query57	1198	1163	1135	1135
query58	254	261	243	243
query59	2737	2864	2739	2739
query60	289	279	296	279
query61	131	133	135	133
query62	750	765	672	672
query63	221	179	180	179
query64	1846	1114	684	684
query65	4411	4420	4218	4218
query66	699	398	305	305
query67	15983	15558	15439	15439
query68	6614	926	509	509
query69	542	303	256	256
query70	1123	1031	1114	1031
query71	503	292	268	268
query72	5932	4848	4915	4848
query73	1437	704	356	356
query74	8844	9030	8600	8600
query75	3991	3240	2734	2734
query76	4312	1197	779	779
query77	686	366	281	281
query78	9919	10061	9280	9280
query79	2336	812	564	564
query80	640	510	438	438
query81	489	265	220	220
query82	453	122	91	91
query83	264	173	160	160
query84	300	90	72	72
query85	762	371	419	371
query86	368	301	282	282
query87	4434	4473	4342	4342
query88	3553	2247	2246	2246
query89	414	312	281	281
query90	1914	202	210	202
query91	226	142	110	110
query92	79	61	57	57
query93	1896	1067	582	582
query94	650	420	310	310
query95	348	267	264	264
query96	486	586	274	274
query97	3158	3238	3099	3099
query98	222	215	203	203
query99	1435	1368	1255	1255
Total cold run time: 297255 ms
Total hot run time: 192044 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.66 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b9ecb1be58c4a6dfecd2cb577548619b194196b3, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.10
query3	0.25	0.19	0.20
query4	1.59	0.18	0.19
query5	0.60	0.57	0.60
query6	1.18	0.71	0.71
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.57	0.53	0.51
query10	0.60	0.60	0.57
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.60	0.59	0.60
query14	2.67	2.69	2.68
query15	0.90	0.85	0.84
query16	0.38	0.38	0.38
query17	1.04	1.00	1.05
query18	0.21	0.20	0.20
query19	1.84	1.83	1.96
query20	0.02	0.01	0.01
query21	15.37	0.90	0.55
query22	0.74	1.08	0.70
query23	14.94	1.40	0.64
query24	6.74	1.85	0.38
query25	0.31	0.24	0.15
query26	0.65	0.16	0.13
query27	0.05	0.05	0.05
query28	9.28	0.91	0.42
query29	12.52	3.99	3.29
query30	0.25	0.09	0.06
query31	2.84	0.58	0.38
query32	3.23	0.55	0.47
query33	3.04	3.03	3.07
query34	15.86	5.14	4.47
query35	4.55	4.52	4.51
query36	0.68	0.49	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.02
query40	0.16	0.13	0.13
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.5 s
Total hot run time: 30.66 s

starocean999
starocean999 previously approved these changes Mar 28, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 28, 2025
@LiBinfeng-01
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 29, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 34067 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9801bab77b34dca1b474009541b506725e28a3d2, data reload: false

------ Round 1 ----------------------------------
q1	25766	5061	4979	4979
q2	2066	303	169	169
q3	10398	1243	669	669
q4	10220	1013	544	544
q5	7526	2345	2348	2345
q6	196	165	133	133
q7	906	744	623	623
q8	9326	1289	1078	1078
q9	6784	5171	5166	5166
q10	6866	2277	1890	1890
q11	493	285	273	273
q12	357	352	220	220
q13	17766	3662	3047	3047
q14	227	220	202	202
q15	536	472	486	472
q16	628	630	581	581
q17	588	854	368	368
q18	7540	7249	7097	7097
q19	1645	943	562	562
q20	334	330	190	190
q21	4017	2601	2476	2476
q22	1085	1027	983	983
Total cold run time: 115270 ms
Total hot run time: 34067 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5185	5094	5094	5094
q2	263	316	227	227
q3	2151	2640	2233	2233
q4	1373	1800	1361	1361
q5	4476	4359	4359	4359
q6	220	174	131	131
q7	1985	1879	1797	1797
q8	2536	2550	2497	2497
q9	7341	7315	7033	7033
q10	2990	3205	2744	2744
q11	562	495	498	495
q12	682	740	598	598
q13	3460	3836	3334	3334
q14	316	306	268	268
q15	520	477	482	477
q16	664	686	640	640
q17	1160	1576	1372	1372
q18	7750	7481	7350	7350
q19	825	866	1044	866
q20	1937	1986	1830	1830
q21	5397	4911	4825	4825
q22	1096	1046	1038	1038
Total cold run time: 52889 ms
Total hot run time: 50569 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c83459c5d7a308837a084932a9b59a2decbea8ca, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.19
query5	0.59	0.59	0.62
query6	1.20	0.71	0.71
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.53	0.52
query10	0.57	0.58	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.59	0.61
query14	2.67	2.71	2.70
query15	0.92	0.86	0.85
query16	0.39	0.37	0.38
query17	1.02	1.01	1.02
query18	0.21	0.20	0.20
query19	1.83	1.83	1.90
query20	0.01	0.01	0.01
query21	15.35	0.89	0.55
query22	0.76	1.21	0.99
query23	14.72	1.38	0.60
query24	6.90	1.37	1.63
query25	0.51	0.23	0.12
query26	0.60	0.16	0.13
query27	0.04	0.04	0.04
query28	10.64	0.88	0.44
query29	12.55	4.02	3.36
query30	0.26	0.09	0.06
query31	2.83	0.59	0.39
query32	3.24	0.54	0.47
query33	3.04	3.06	3.12
query34	15.69	5.13	4.54
query35	4.51	4.48	4.50
query36	0.66	0.50	0.49
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.18	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.76 s
Total hot run time: 32.13 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 1, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 1, 2025

PR approved by at least one committer and no changes requested.

@starocean999 starocean999 merged commit e373ad5 into apache:master Apr 1, 2025
24 of 26 checks passed
LiBinfeng-01 pushed a commit to LiBinfeng-01/doris that referenced this pull request Apr 1, 2025
LiBinfeng-01 pushed a commit to LiBinfeng-01/doris that referenced this pull request Apr 1, 2025
LiBinfeng-01 pushed a commit to LiBinfeng-01/doris that referenced this pull request Apr 16, 2025
LiBinfeng-01 pushed a commit to LiBinfeng-01/doris that referenced this pull request Apr 16, 2025
LiBinfeng-01 pushed a commit to LiBinfeng-01/doris that referenced this pull request Apr 24, 2025
LiBinfeng-01 pushed a commit to LiBinfeng-01/doris that referenced this pull request Apr 24, 2025
dataroaring pushed a commit that referenced this pull request Apr 25, 2025
Jibing-Li pushed a commit to Jibing-Li/incubator-doris that referenced this pull request May 7, 2025
Jibing-Li pushed a commit to Jibing-Li/incubator-doris that referenced this pull request May 8, 2025
Jibing-Li pushed a commit to Jibing-Li/incubator-doris that referenced this pull request May 8, 2025
yiguolei pushed a commit that referenced this pull request May 8, 2025
…fix code point count (#49575) (#50667)

backport: #49575

Co-authored-by: LiBinfeng <libinfeng@selectdb.com>
@yiguolei yiguolei mentioned this pull request May 13, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.6-merged p0_w reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants