Skip to content

branch-3.0: [fix](function) fix error result when input utf8 in url_encode, strright, append_trailing_char_if_absent #49127 #50650

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
github-actions wants to merge 1 commit into from
Closed

branch-3.0: [fix](function) fix error result when input utf8 in url_encode, strright, append_trailing_char_if_absent #49127 #50650

github-actions wants to merge 1 commit into from

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented May 7, 2025

Cherry-picked from #49127

@Mryange
[fix](function) fix error result when input utf8 in url_encode, strri…
5badf38 
…ght, append_trailing_char_if_absent (#49127)

### What problem does this PR solve?
 
The url_encode function previously performed a modulus operation on a
signed number. Converting it to an unsigned number will fix the issue.
```
before
mysql> select url_encode('编码');
+----------------------+
| url_encode('编码')   |
+----------------------+
| %5.%23%0-%5.%10%/(   |
+----------------------+
now
mysql> select url_encode('编码');
+----------------------+
| url_encode('编码')   |
+----------------------+
| %E7%BC%96%E7%A0%81   |
+----------------------+
```

The strright function did not calculate the length according to the
number of UTF-8 characters.
```
before
mysql> select strright("你好世界",5);
+----------------------------+
| strright("你好世界",5)     |
+----------------------------+
|                            |
+----------------------------+
now

mysql> select strright("你好世界",5);
+----------------------------+
| strright("你好世界",5)     |
+----------------------------+
| 你好世界                   |
+----------------------------+
```

he case of inputting a UTF-8 character was not considered.
```
mysql> select append_trailing_char_if_absent('中文', '文');
+-------------------------------------------------+
| append_trailing_char_if_absent('中文', '文')    |
+-------------------------------------------------+
| NULL                                            |
+-------------------------------------------------+
now
mysql> select append_trailing_char_if_absent('中文', '文');
+-------------------------------------------------+
| append_trailing_char_if_absent('中文', '文')    |
+-------------------------------------------------+
| 中文                                            |
+-------------------------------------------------+
```

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@github-actions github-actions bot requested a review from dataroaring as a code owner May 7, 2025 07:10
@hello-stephen
Copy link
Contributor

hello-stephen commented May 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this May 7, 2025
@dataroaring dataroaring reopened this May 7, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented May 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 7, 2025

TPC-H: Total hot run time: 40114 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5badf38e6a053f4d21a10aae13a786f803329054, data reload: false

------ Round 1 ----------------------------------
q1	17571	6882	6608	6608
q2	2078	167	163	163
q3	10642	1094	1159	1094
q4	10558	736	706	706
q5	7760	2875	2848	2848
q6	224	135	132	132
q7	977	624	608	608
q8	9364	1934	2029	1934
q9	6599	6388	6410	6388
q10	7048	2297	2313	2297
q11	464	260	266	260
q12	397	209	206	206
q13	17780	2963	2977	2963
q14	235	220	206	206
q15	510	472	477	472
q16	683	580	580	580
q17	977	564	519	519
q18	7312	6668	6648	6648
q19	1401	1073	1063	1063
q20	475	205	198	198
q21	4059	3251	3222	3222
q22	1105	1016	999	999
Total cold run time: 108219 ms
Total hot run time: 40114 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6628	6583	6659	6583
q2	336	231	235	231
q3	2937	2761	2927	2761
q4	2056	1799	1746	1746
q5	5785	5742	5748	5742
q6	214	128	130	128
q7	2215	1803	1835	1803
q8	3402	3580	3546	3546
q9	8989	8829	8934	8829
q10	3588	3543	3555	3543
q11	587	502	511	502
q12	826	636	623	623
q13	9612	3151	3191	3151
q14	294	262	271	262
q15	517	490	474	474
q16	704	645	658	645
q17	1879	1643	1620	1620
q18	8237	7674	7791	7674
q19	1709	1673	1576	1576
q20	2066	1838	1790	1790
q21	5428	5318	5360	5318
q22	1155	1069	1014	1014
Total cold run time: 69164 ms
Total hot run time: 59561 ms

@doris-robot
Copy link

doris-robot commented May 7, 2025

TPC-DS: Total hot run time: 197440 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5badf38e6a053f4d21a10aae13a786f803329054, data reload: false

query1	1307	918	914	914
query2	6288	2002	1958	1958
query3	10910	4498	4494	4494
query4	61939	29287	23575	23575
query5	5223	465	445	445
query6	390	179	171	171
query7	5439	314	303	303
query8	302	226	221	221
query9	8463	2598	2583	2583
query10	444	272	259	259
query11	17827	15137	15765	15137
query12	166	115	109	109
query13	1420	449	424	424
query14	10424	7736	7172	7172
query15	200	184	173	173
query16	7198	482	484	482
query17	1180	597	590	590
query18	1871	336	312	312
query19	208	164	155	155
query20	122	112	109	109
query21	207	104	106	104
query22	4804	4840	4608	4608
query23	34547	34135	33876	33876
query24	6213	2906	2900	2900
query25	497	394	408	394
query26	659	165	179	165
query27	1993	353	345	345
query28	4231	2450	2416	2416
query29	684	459	418	418
query30	250	157	168	157
query31	995	852	816	816
query32	70	53	60	53
query33	461	283	289	283
query34	936	518	522	518
query35	861	770	711	711
query36	1083	954	947	947
query37	112	62	64	62
query38	4087	3963	3965	3963
query39	1517	1500	1493	1493
query40	202	99	97	97
query41	49	46	47	46
query42	112	98	97	97
query43	524	487	478	478
query44	1173	805	803	803
query45	180	164	169	164
query46	1161	774	719	719
query47	2032	1896	1953	1896
query48	478	388	378	378
query49	730	398	399	398
query50	863	430	429	429
query51	7402	7286	7356	7286
query52	103	94	93	93
query53	268	184	184	184
query54	592	453	479	453
query55	81	78	86	78
query56	279	246	254	246
query57	1318	1212	1153	1153
query58	225	221	213	213
query59	3177	3190	2980	2980
query60	278	236	245	236
query61	114	172	108	108
query62	758	686	664	664
query63	217	189	186	186
query64	1411	665	637	637
query65	3303	3223	3206	3206
query66	699	302	302	302
query67	16065	15608	15484	15484
query68	3729	581	575	575
query69	412	266	259	259
query70	1157	1157	1114	1114
query71	349	265	255	255
query72	6373	4026	3985	3985
query73	741	342	360	342
query74	10343	9235	9078	9078
query75	3352	2675	2696	2675
query76	1968	1135	1036	1036
query77	506	280	280	280
query78	10547	9604	9571	9571
query79	1468	599	592	592
query80	869	446	475	446
query81	513	246	236	236
query82	1035	95	94	94
query83	167	149	148	148
query84	291	86	81	81
query85	875	319	299	299
query86	329	300	285	285
query87	4415	4264	4183	4183
query88	3759	2392	2358	2358
query89	410	295	296	295
query90	2006	188	189	188
query91	176	146	154	146
query92	60	49	53	49
query93	1655	552	558	552
query94	715	289	294	289
query95	355	256	262	256
query96	616	279	282	279
query97	3340	3162	3167	3162
query98	213	201	192	192
query99	1562	1311	1346	1311
Total cold run time: 313628 ms
Total hot run time: 197440 ms

@doris-robot
Copy link

doris-robot commented May 7, 2025

ClickBench: Total hot run time: 31.87 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5badf38e6a053f4d21a10aae13a786f803329054, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.52	0.49	0.52
query6	1.13	0.73	0.73
query7	0.02	0.04	0.02
query8	0.04	0.03	0.03
query9	0.56	0.51	0.51
query10	0.56	0.56	0.55
query11	0.16	0.11	0.11
query12	0.14	0.10	0.11
query13	0.60	0.59	0.58
query14	2.69	2.81	2.86
query15	0.89	0.82	0.83
query16	0.39	0.41	0.38
query17	1.06	1.05	1.03
query18	0.23	0.22	0.22
query19	1.91	1.79	2.04
query20	0.01	0.01	0.01
query21	15.36	0.59	0.58
query22	2.26	1.87	2.80
query23	16.92	1.00	0.90
query24	3.67	0.89	0.44
query25	0.17	0.16	0.05
query26	0.44	0.12	0.13
query27	0.04	0.05	0.04
query28	10.73	0.51	0.51
query29	12.58	3.30	3.24
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.22	0.49	0.45
query33	2.98	3.02	3.02
query34	17.16	4.50	4.54
query35	4.57	4.53	4.59
query36	0.66	0.48	0.48
query37	0.08	0.05	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 107.23 s
Total hot run time: 31.87 s

@Mryange
Copy link
Contributor

Mryange commented May 7, 2025

#50659

@dataroaring dataroaring closed this May 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring Awaiting requested review from dataroaring

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen @doris-robot @Mryange @dataroaring