Skip to content

Conversation

@Jibing-Li
Copy link
Contributor

backport: #50085

…pache#50085)

Use utf-8 when convert string like literal to double.
StringLike columns in Doris are all stored with utf-8 encoding. So we
need to use utf-8 encoding to read the column statistics min/max value.
Otherwise, Java will use the system default encoding. In this case,
doris may read wrong statistics min/max value.
@Jibing-Li
Copy link
Contributor Author

run buildall

@Thearas
Copy link
Contributor

Thearas commented Apr 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Jibing-Li Jibing-Li marked this pull request as ready for review April 18, 2025 02:09
@Jibing-Li Jibing-Li requested a review from dataroaring as a code owner April 18, 2025 02:09
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40436 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ddcf1a2f849707ab04668acd53b0aa6f024727e4, data reload: false

------ Round 1 ----------------------------------
q1	17567	6871	6683	6683
q2	2078	183	171	171
q3	10684	1083	1176	1083
q4	10434	761	715	715
q5	7764	2810	2861	2810
q6	217	137	140	137
q7	983	613	617	613
q8	9369	1963	2013	1963
q9	6619	6461	6456	6456
q10	7040	2244	2351	2244
q11	476	271	263	263
q12	398	223	214	214
q13	17787	2981	2981	2981
q14	232	213	215	213
q15	509	462	482	462
q16	676	601	587	587
q17	992	601	581	581
q18	7214	6763	6744	6744
q19	1408	1129	1009	1009
q20	486	212	210	210
q21	4073	3311	3436	3311
q22	1122	1001	986	986
Total cold run time: 108128 ms
Total hot run time: 40436 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6686	6652	6592	6592
q2	332	236	231	231
q3	2897	2805	2962	2805
q4	2085	1848	1837	1837
q5	5780	5734	5741	5734
q6	212	129	131	129
q7	2235	1840	1808	1808
q8	3412	3608	3535	3535
q9	8998	8894	8969	8894
q10	3584	3552	3563	3552
q11	608	500	513	500
q12	797	623	603	603
q13	9880	3206	3201	3201
q14	300	262	271	262
q15	526	469	465	465
q16	687	673	675	673
q17	1858	1640	1659	1640
q18	8228	7758	7742	7742
q19	1702	1449	1604	1449
q20	2072	1832	1853	1832
q21	5587	5364	5478	5364
q22	1146	1051	1035	1035
Total cold run time: 69612 ms
Total hot run time: 59883 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197844 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ddcf1a2f849707ab04668acd53b0aa6f024727e4, data reload: false

query1	1284	921	894	894
query2	6273	2100	2075	2075
query3	11015	4650	4526	4526
query4	61376	29872	23640	23640
query5	5272	463	451	451
query6	402	173	182	173
query7	5452	320	313	313
query8	311	226	219	219
query9	8563	2599	2602	2599
query10	456	271	263	263
query11	17532	15067	15763	15067
query12	160	100	100	100
query13	1421	441	440	440
query14	9870	7579	6784	6784
query15	203	185	182	182
query16	7114	473	484	473
query17	1151	602	599	599
query18	1853	347	321	321
query19	204	167	156	156
query20	122	110	116	110
query21	220	109	115	109
query22	4722	4439	4641	4439
query23	34539	34505	34311	34311
query24	6417	2931	2929	2929
query25	534	445	430	430
query26	662	174	172	172
query27	1939	354	355	354
query28	4080	2524	2484	2484
query29	730	475	421	421
query30	252	162	170	162
query31	1005	822	823	822
query32	69	55	57	55
query33	468	289	285	285
query34	926	510	536	510
query35	875	772	750	750
query36	1111	954	997	954
query37	117	69	69	69
query38	4121	3999	3945	3945
query39	1505	1487	1502	1487
query40	220	102	96	96
query41	53	49	48	48
query42	111	98	103	98
query43	525	505	478	478
query44	1173	815	818	815
query45	186	169	165	165
query46	1149	741	749	741
query47	2046	1911	1908	1908
query48	489	392	403	392
query49	725	401	392	392
query50	850	428	429	428
query51	7336	7183	7243	7183
query52	106	90	88	88
query53	263	184	190	184
query54	574	458	486	458
query55	81	78	81	78
query56	275	237	258	237
query57	1279	1168	1130	1130
query58	239	222	216	216
query59	3250	3066	3134	3066
query60	278	257	259	257
query61	114	105	141	105
query62	775	701	700	700
query63	230	186	192	186
query64	1432	706	691	691
query65	3251	3215	3195	3195
query66	707	302	295	295
query67	15735	15557	15421	15421
query68	4182	588	573	573
query69	440	274	267	267
query70	1207	1073	1131	1073
query71	344	254	259	254
query72	6371	4018	4049	4018
query73	754	346	349	346
query74	10180	9294	9161	9161
query75	3337	2667	2639	2639
query76	2069	1083	1079	1079
query77	539	274	281	274
query78	10588	9554	9651	9554
query79	1930	607	592	592
query80	1491	439	422	422
query81	539	237	240	237
query82	1298	91	88	88
query83	267	147	141	141
query84	285	77	74	74
query85	1080	305	295	295
query86	376	308	299	299
query87	4471	4206	4248	4206
query88	3997	2389	2372	2372
query89	443	307	301	301
query90	2029	187	189	187
query91	181	152	152	152
query92	70	49	50	49
query93	1974	556	563	556
query94	816	299	298	298
query95	355	272	260	260
query96	624	281	284	281
query97	3297	3186	3170	3170
query98	212	202	193	193
query99	1580	1317	1309	1309
Total cold run time: 315104 ms
Total hot run time: 197844 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.29 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ddcf1a2f849707ab04668acd53b0aa6f024727e4, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.02
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.53	0.51	0.51
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.53	0.55	0.56
query11	0.15	0.11	0.10
query12	0.13	0.11	0.12
query13	0.62	0.59	0.59
query14	2.77	2.82	2.72
query15	0.89	0.82	0.82
query16	0.40	0.38	0.37
query17	0.94	1.07	1.07
query18	0.24	0.22	0.22
query19	1.96	1.88	2.05
query20	0.01	0.02	0.01
query21	15.36	0.57	0.57
query22	2.96	2.41	2.46
query23	17.01	1.00	0.71
query24	2.65	1.14	0.50
query25	0.33	0.14	0.04
query26	0.31	0.13	0.12
query27	0.04	0.05	0.04
query28	11.32	0.48	0.46
query29	12.61	3.21	3.19
query30	0.25	0.06	0.06
query31	2.87	0.40	0.38
query32	3.23	0.45	0.46
query33	2.95	3.00	2.98
query34	17.33	4.57	4.58
query35	4.57	4.60	4.59
query36	0.66	0.50	0.48
query37	0.09	0.06	0.07
query38	0.04	0.03	0.04
query39	0.04	0.02	0.03
query40	0.17	0.14	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.82 s
Total hot run time: 32.29 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 39e26d2 into apache:branch-3.0 Apr 22, 2025
23 of 24 checks passed
@Jibing-Li Jibing-Li deleted the utf83.0 branch April 24, 2025 02:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants