Skip to content

[fix] (Nereids) fix date function rewrite on datetimev1 column bug #32569

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 4 commits into from
Mar 23, 2024
Merged

[fix] (Nereids) fix date function rewrite on datetimev1 column bug #32569

yiguolei merged 4 commits into from
Mar 23, 2024

Conversation

@ziyanTOP
Copy link
Contributor

@ziyanTOP ziyanTOP commented Mar 20, 2024

Proposed changes

Issue Number: close #32329

Avoid '24:00:00' time literal for datetimev1 column after handling by date function rewrite in nereids planner

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Mar 20, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@ziyanTOP ziyanTOP force-pushed the datetimev1_fix branch 2 times, most recently from 2fac725 to 006df96 Compare March 21, 2024 03:20
@ziyanTOP
Copy link
Contributor Author

ziyanTOP commented Mar 21, 2024

run buildall

@ziyanTOP ziyanTOP requested a review from yiguolei March 21, 2024 10:03
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 22, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 22, 2024

PR approved by anyone and no changes requested.

@924060929
Copy link
Contributor

924060929 commented Mar 22, 2024

@ziyanTOP good job👍🏻, would you submit a fix pr for branch-2.0?

@924060929
Copy link
Contributor

924060929 commented Mar 22, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 22, 2024

TPC-H: Total hot run time: 38474 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 912a6c0d9ed5f09b602a2d102011090864b530e2, data reload: false

------ Round 1 ----------------------------------
q1	17778	4906	4159	4159
q2	2113	162	168	162
q3	10884	1158	1247	1158
q4	10514	864	771	771
q5	7890	3067	3030	3030
q6	206	132	122	122
q7	1075	596	591	591
q8	9901	2047	2055	2047
q9	7390	6771	6695	6695
q10	9643	3488	3546	3488
q11	439	221	219	219
q12	389	196	192	192
q13	17809	2873	2879	2873
q14	239	210	214	210
q15	503	481	455	455
q16	491	377	371	371
q17	977	563	588	563
q18	7257	6557	6581	6557
q19	1556	1421	1507	1421
q20	551	277	257	257
q21	3532	2945	2837	2837
q22	339	296	296	296
Total cold run time: 111476 ms
Total hot run time: 38474 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4115	4098	4054	4054
q2	330	240	237	237
q3	2953	2883	2834	2834
q4	1879	1596	1581	1581
q5	5303	5356	5378	5356
q6	194	116	117	116
q7	2284	1885	1837	1837
q8	3151	3337	3274	3274
q9	8732	8715	8723	8715
q10	3777	3792	3781	3781
q11	541	448	439	439
q12	734	526	542	526
q13	16501	2874	2865	2865
q14	293	258	263	258
q15	517	470	472	470
q16	498	427	445	427
q17	1780	1548	1515	1515
q18	7677	7176	7075	7075
q19	1633	1583	1543	1543
q20	1920	1709	1717	1709
q21	4855	4826	4820	4820
q22	538	458	463	458
Total cold run time: 70205 ms
Total hot run time: 53890 ms

@doris-robot
Copy link

doris-robot commented Mar 22, 2024

TPC-DS: Total hot run time: 186861 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 912a6c0d9ed5f09b602a2d102011090864b530e2, data reload: false

query1	945	385	356	356
query2	7335	2081	2011	2011
query3	6704	213	222	213
query4	31840	21284	21387	21284
query5	4334	401	393	393
query6	271	182	177	177
query7	4633	290	293	290
query8	241	168	179	168
query9	9340	2289	2289	2289
query10	560	253	259	253
query11	14886	14568	14572	14568
query12	143	87	87	87
query13	1635	421	418	418
query14	11610	10907	11035	10907
query15	270	207	203	203
query16	8070	259	260	259
query17	1957	583	552	552
query18	2086	291	285	285
query19	279	157	159	157
query20	95	87	87	87
query21	203	128	128	128
query22	5055	4883	4777	4777
query23	33563	33050	32887	32887
query24	11618	2873	2824	2824
query25	665	389	394	389
query26	1844	161	154	154
query27	3077	353	359	353
query28	7663	1871	1857	1857
query29	1007	654	636	636
query30	306	147	149	147
query31	946	743	748	743
query32	88	59	58	58
query33	776	269	259	259
query34	1061	473	486	473
query35	840	608	603	603
query36	1008	891	880	880
query37	154	82	83	82
query38	3591	3452	3477	3452
query39	1511	1434	1406	1406
query40	303	114	113	113
query41	51	47	47	47
query42	104	93	95	93
query43	482	449	453	449
query44	1117	733	726	726
query45	294	259	271	259
query46	1115	725	688	688
query47	1925	1851	1866	1851
query48	443	357	353	353
query49	1229	339	363	339
query50	771	365	372	365
query51	6677	6597	6618	6597
query52	109	93	91	91
query53	354	277	284	277
query54	331	234	246	234
query55	84	79	76	76
query56	245	233	236	233
query57	1227	1130	1159	1130
query58	241	214	216	214
query59	2975	2670	2591	2591
query60	329	250	254	250
query61	116	112	115	112
query62	642	442	442	442
query63	307	280	280	280
query64	6874	3917	4087	3917
query65	3125	3023	3050	3023
query66	1414	378	353	353
query67	15570	14833	15115	14833
query68	10166	539	532	532
query69	689	383	374	374
query70	1350	1164	1101	1101
query71	549	274	271	271
query72	6677	2675	2542	2542
query73	1586	315	316	315
query74	7314	6678	6780	6678
query75	5172	2907	2916	2907
query76	5952	898	842	842
query77	637	280	261	261
query78	11048	10247	10113	10113
query79	11595	533	507	507
query80	1705	383	370	370
query81	499	225	217	217
query82	342	198	205	198
query83	216	141	142	141
query84	287	78	76	76
query85	1183	319	312	312
query86	357	320	305	305
query87	3733	3532	3550	3532
query88	5261	2287	2290	2287
query89	477	365	363	363
query90	2040	170	171	170
query91	163	136	136	136
query92	61	49	47	47
query93	6410	504	481	481
query94	1327	171	175	171
query95	452	328	331	328
query96	597	268	266	266
query97	3055	2882	2920	2882
query98	230	216	217	216
query99	1129	942	919	919
Total cold run time: 322923 ms
Total hot run time: 186861 ms

@doris-robot
Copy link

doris-robot commented Mar 22, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 912a6c0d9ed5f09b602a2d102011090864b530e2 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       20.7 seconds inserted 10000000 Rows, about 483K ops/s

@ziyanTOP
Copy link
Contributor Author

ziyanTOP commented Mar 23, 2024

@ziyanTOP good job👍🏻, would you submit a fix pr for branch-2.0?

ok, I will

@ziyanTOP ziyanTOP mentioned this pull request Mar 23, 2024
Merged
@ziyanTOP
Copy link
Contributor Author

ziyanTOP commented Mar 23, 2024

done @924060929

@yiguolei yiguolei merged commit ba85b02 into apache:master Mar 23, 2024
yiguolei pushed a commit that referenced this pull request Mar 24, 2024
@ziyanTOP @Doris-Extras
[fix] (Nereids) fix date function rewrite on datetimev1 column bug (#…
b9788e5 
…32569)
924060929 pushed a commit that referenced this pull request Mar 25, 2024
@ziyanTOP
[fix](Nereids) fix date function rewrite on datetimev1 column bug #32569
f0839fb 
 (#32719)

cherry picked from #32569
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@ziyanTOP
[fix](Nereids) fix date function rewrite on datetimev1 column bug apa…
f979af9 
…che#32569 (apache#32719)

cherry picked from apache#32569
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929 924060929 924060929 approved these changes

@yiguolei yiguolei yiguolei approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.8-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] [INVALID_ARGUMENT]Create Expr failed because [E33] Invalid value: 2024-03-11 24:00:00 for type DateTime

7 participants

@ziyanTOP @doris-robot @924060929 @yiguolei @morrySnow @xiaokang @wm1581066