Skip to content

[fix](datetime) fix datetime round on BE (#31205) #31229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 1 commit into from
Feb 21, 2024
Merged

[fix](datetime) fix datetime round on BE (#31205) #31229

yiguolei merged 1 commit into from
Feb 21, 2024

Conversation

@zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Feb 21, 2024 •
edited
Loading 

with tmp as (
            select CONCAT(
                YEAR('2024-02-06 03:37:07.157'), '-', 
                LPAD(MONTH('2024-02-06 03:37:07.157'), 2, '0'), '-',
                LPAD(DAY('2024-02-06 03:37:07.157'), 2, '0'), ' ',
                LPAD(HOUR('2024-02-06 03:37:07.157'), 2, '0'), ':',
                LPAD(MINUTE('2024-02-06 03:37:07.157'), 2, '0'), ':',
                LPAD(SECOND('2024-02-06 03:37:07.157'), 2, '0'), '.', "123456789" )
            AS generated_string)
            select generated_string, cast(generated_string as DateTime(6)) from tmp
before (incorrect round)

+-------------------------------+-----------------------------------------+
| generated_string              | cast(generated_string as DATETIMEV2(6)) |
+-------------------------------+-----------------------------------------+
| 2024-02-06 03:37:07.123456789 | 2024-02-06 03:37:07.123456              |
+-------------------------------+-----------------------------------------+

after (round up, keep consistent with mysql):

+-------------------------------+-----------------------------------------+
| generated_string              | cast(generated_string as DATETIMEV2(6)) |
+-------------------------------+-----------------------------------------+
| 2024-02-06 03:37:07.123456789 | 2024-02-06 03:37:07.123457              |
+-------------------------------+-----------------------------------------+
1 row in set (0.03 sec)

same work with #30744 but implemented on BE

pick #31205

@zhiqiang-hhhh
[fix](datetime) fix datetime round on BE (apache#31205)
31dad31 
with tmp as (
            select CONCAT(
                YEAR('2024-02-06 03:37:07.157'), '-', 
                LPAD(MONTH('2024-02-06 03:37:07.157'), 2, '0'), '-',
                LPAD(DAY('2024-02-06 03:37:07.157'), 2, '0'), ' ',
                LPAD(HOUR('2024-02-06 03:37:07.157'), 2, '0'), ':',
                LPAD(MINUTE('2024-02-06 03:37:07.157'), 2, '0'), ':',
                LPAD(SECOND('2024-02-06 03:37:07.157'), 2, '0'), '.', "123456789" )
            AS generated_string)
            select generated_string, cast(generated_string as DateTime(6)) from tmp
before (incorrect round)

+-------------------------------+-----------------------------------------+
| generated_string              | cast(generated_string as DATETIMEV2(6)) |
+-------------------------------+-----------------------------------------+
| 2024-02-06 03:37:07.123456789 | 2024-02-06 03:37:07.123456              |
+-------------------------------+-----------------------------------------+
after (round up, keep consistent with mysql):

+-------------------------------+-----------------------------------------+
| generated_string              | cast(generated_string as DATETIMEV2(6)) |
+-------------------------------+-----------------------------------------+
| 2024-02-06 03:37:07.123456789 | 2024-02-06 03:37:07.123457              |
+-------------------------------+-----------------------------------------+
1 row in set (0.03 sec)
same work with apache#30744 but implemented on BE
@doris-robot
Copy link

doris-robot commented Feb 21, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@zhiqiang-hhhh
Copy link
Contributor Author

zhiqiang-hhhh commented Feb 21, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Feb 21, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Feb 21, 2024

TPC-H: Total hot run time: 50272 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 31dad314e5d8ef46a0184b729f5ef31cbf20a26a, data reload: false

------ Round 1 ----------------------------------
q1	17616	4521	4438	4438
q2	2047	148	151	148
q3	10473	1900	1903	1900
q4	10170	1246	1319	1246
q5	8692	4056	4007	4007
q6	230	125	126	125
q7	2063	1620	1634	1620
q8	9289	2720	2714	2714
q9	11186	10679	10699	10679
q10	8626	3531	3528	3528
q11	413	242	248	242
q12	475	305	308	305
q13	18515	3939	3998	3939
q14	359	327	317	317
q15	512	448	467	448
q16	704	585	583	583
q17	1128	979	965	965
q18	7217	6909	6752	6752
q19	1674	1552	1511	1511
q20	540	301	320	301
q21	4451	4108	4140	4108
q22	506	396	406	396
Total cold run time: 116886 ms
Total hot run time: 50272 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4334	4308	4308	4308
q2	316	229	218	218
q3	4161	4155	4183	4155
q4	2749	2749	2754	2749
q5	7312	7258	7223	7223
q6	234	121	118	118
q7	3218	2837	2866	2837
q8	4309	4413	4417	4413
q9	17639	17379	17269	17269
q10	4217	4273	4294	4273
q11	743	663	679	663
q12	1025	847	857	847
q13	7177	3769	3746	3746
q14	452	416	418	416
q15	506	463	445	445
q16	753	705	696	696
q17	3789	3856	3814	3814
q18	8902	8706	8729	8706
q19	1731	1708	1667	1667
q20	2382	2167	2112	2112
q21	8460	8423	8620	8423
q22	1009	980	958	958
Total cold run time: 85418 ms
Total hot run time: 80056 ms

@doris-robot
Copy link

doris-robot commented Feb 21, 2024

TeamCity be ut coverage result:
Function Coverage: 37.80% (8035/21254)
Line Coverage: 29.49% (65542/222227)
Region Coverage: 28.96% (33713/116428)
Branch Coverage: 24.81% (17302/69732)
Coverage Report: http://coverage.selectdb-in.cc/coverage/31dad314e5d8ef46a0184b729f5ef31cbf20a26a_31dad314e5d8ef46a0184b729f5ef31cbf20a26a/report/index.html

@doris-robot
Copy link

doris-robot commented Feb 21, 2024

TPC-DS: Total hot run time: 241641 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 31dad314e5d8ef46a0184b729f5ef31cbf20a26a, data reload: false

query1	923	385	379	379
query2	6549	2254	2069	2069
query3	6905	200	198	198
query4	20195	18011	18008	18008
query5	19723	6572	6501	6501
query6	287	212	231	212
query7	4148	296	295	295
query8	259	238	238	238
query9	3111	2678	2628	2628
query10	428	307	305	305
query11	11366	10669	10610	10610
query12	113	79	75	75
query13	5580	643	620	620
query14	17547	13506	13364	13364
query15	361	238	236	236
query16	6427	270	266	266
query17	1733	1443	846	846
query18	2335	402	404	402
query19	196	151	146	146
query20	78	79	74	74
query21	189	90	102	90
query22	5349	5259	5172	5172
query23	32599	32049	31896	31896
query24	6936	6497	6498	6497
query25	514	428	422	422
query26	532	161	162	161
query27	1873	289	289	289
query28	6053	2246	2204	2204
query29	2865	2812	2721	2721
query30	237	162	158	158
query31	888	735	732	732
query32	69	59	58	58
query33	399	238	251	238
query34	838	463	463	463
query35	1149	912	929	912
query36	1740	1619	1899	1619
query37	89	57	60	57
query38	3091	2959	2908	2908
query39	1370	1325	1309	1309
query40	204	92	95	92
query41	34	32	35	32
query42	88	85	86	85
query43	647	592	612	592
query44	1137	714	709	709
query45	242	227	227	227
query46	1253	983	973	973
query47	1789	1713	1756	1713
query48	972	684	654	654
query49	616	368	365	365
query50	862	639	637	637
query51	4759	4712	4651	4651
query52	97	81	77	77
query53	443	322	322	322
query54	2675	2463	2456	2456
query55	86	73	79	73
query56	199	196	208	196
query57	1178	1200	1064	1064
query58	207	198	200	198
query59	3516	3191	3301	3191
query60	211	189	185	185
query61	84	83	81	81
query62	782	541	546	541
query63	480	332	339	332
query64	2392	1479	1288	1288
query65	3593	3604	3592	3592
query66	772	375	366	366
query67	16661	15522	15430	15430
query68	7456	653	653	653
query69	573	344	337	337
query70	1725	1724	1665	1665
query71	409	292	305	292
query72	6514	3407	3424	3407
query73	726	312	308	308
query74	6313	5795	5831	5795
query75	4736	3710	3636	3636
query76	4393	1165	1223	1165
query77	521	259	257	257
query78	31406	51074	51397	51074
query79	15402	623	644	623
query80	4958	374	382	374
query81	560	229	228	228
query82	1432	96	94	94
query83	324	131	131	131
query84	260	70	69	69
query85	1913	287	284	284
query86	460	367	368	367
query87	3253	2963	2966	2963
query88	7019	2328	2302	2302
query89	440	278	280	278
query90	2501	202	206	202
query91	152	115	122	115
query92	67	51	49	49
query93	6312	570	574	570
query94	1747	205	203	203
query95	1098	1065	1054	1054
query96	645	325	321	321
query97	6594	6371	6305	6305
query98	176	168	175	168
query99	3715	852	896	852
Total cold run time: 344823 ms
Total hot run time: 241641 ms

@doris-robot
Copy link

doris-robot commented Feb 21, 2024

ClickBench: Total hot run time: 30.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 31dad314e5d8ef46a0184b729f5ef31cbf20a26a, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.24	0.04	0.05
query4	1.83	0.06	0.07
query5	0.52	0.52	0.52
query6	1.28	0.61	0.62
query7	0.01	0.01	0.01
query8	0.03	0.03	0.03
query9	0.51	0.48	0.48
query10	0.54	0.54	0.53
query11	0.12	0.08	0.09
query12	0.11	0.08	0.09
query13	0.63	0.61	0.61
query14	0.78	0.80	0.78
query15	0.78	0.76	0.75
query16	0.38	0.36	0.39
query17	1.02	1.00	1.00
query18	0.20	0.28	0.24
query19	1.94	1.85	1.84
query20	0.02	0.01	0.00
query21	15.47	0.58	0.56
query22	2.39	2.63	1.29
query23	17.05	1.10	0.96
query24	4.25	1.51	0.96
query25	1.52	0.13	0.12
query26	0.23	0.14	0.14
query27	0.11	0.11	0.10
query28	8.29	0.72	0.71
query29	12.64	2.31	2.32
query30	0.58	0.52	0.48
query31	2.81	0.38	0.39
query32	3.40	0.50	0.49
query33	3.04	3.08	3.07
query34	15.25	4.85	4.81
query35	4.89	4.86	4.88
query36	1.06	1.01	1.01
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.07	0.02	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 104.39 s
Total hot run time: 30.57 s

@doris-robot
Copy link

doris-robot commented Feb 21, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 31dad314e5d8ef46a0184b729f5ef31cbf20a26a with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@yiguolei yiguolei merged commit 37239d8 into apache:branch-2.0 Feb 21, 2024
@zhiqiang-hhhh zhiqiang-hhhh deleted the ch-pick-31205 branch February 21, 2024 11:48
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zhiqiang-hhhh
[fix](datetime) fix datetime round on BE (apache#31205) (apache#31229)
2b6e919 
with tmp as (
            select CONCAT(
                YEAR('2024-02-06 03:37:07.157'), '-', 
                LPAD(MONTH('2024-02-06 03:37:07.157'), 2, '0'), '-',
                LPAD(DAY('2024-02-06 03:37:07.157'), 2, '0'), ' ',
                LPAD(HOUR('2024-02-06 03:37:07.157'), 2, '0'), ':',
                LPAD(MINUTE('2024-02-06 03:37:07.157'), 2, '0'), ':',
                LPAD(SECOND('2024-02-06 03:37:07.157'), 2, '0'), '.', "123456789" )
            AS generated_string)
            select generated_string, cast(generated_string as DateTime(6)) from tmp
before (incorrect round)

+-------------------------------+-----------------------------------------+
| generated_string              | cast(generated_string as DATETIMEV2(6)) |
+-------------------------------+-----------------------------------------+
| 2024-02-06 03:37:07.123456789 | 2024-02-06 03:37:07.123456              |
+-------------------------------+-----------------------------------------+
after (round up, keep consistent with mysql):

+-------------------------------+-----------------------------------------+
| generated_string              | cast(generated_string as DATETIMEV2(6)) |
+-------------------------------+-----------------------------------------+
| 2024-02-06 03:37:07.123456789 | 2024-02-06 03:37:07.123457              |
+-------------------------------+-----------------------------------------+
1 row in set (0.03 sec)
same work with apache#30744 but implemented on BE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zhiqiang-hhhh @doris-robot @yiguolei