Skip to content

Conversation

@jacktengg
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

9999-12-31 23:59:59.999999 for type DateTimeV2(0) will round and overflow, and will result in error Invalid value: 9999-12-31 23:59:59.999999 for type DateTimeV2'

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@jacktengg
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.28% (8733/24753)
Line Coverage: 27.08% (71505/264004)
Region Coverage: 26.33% (37095/140909)
Branch Coverage: 23.23% (18972/81678)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2f072c35369a365b105f1de996d43def0373a918_2f072c35369a365b105f1de996d43def0373a918/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38027 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f072c35369a365b105f1de996d43def0373a918, data reload: false

------ Round 1 ----------------------------------
q1	17672	4299	4131	4131
q2	2025	151	144	144
q3	10594	1158	1210	1158
q4	10234	767	810	767
q5	7460	3022	2961	2961
q6	208	123	124	123
q7	1037	568	561	561
q8	9341	2010	2013	2010
q9	7161	6495	6490	6490
q10	8430	3424	3481	3424
q11	446	226	214	214
q12	405	202	198	198
q13	17787	2844	2846	2844
q14	225	215	204	204
q15	500	468	461	461
q16	458	363	352	352
q17	951	536	620	536
q18	7220	6540	6575	6540
q19	5533	1452	1441	1441
q20	553	260	256	256
q21	3642	2928	2984	2928
q22	348	298	284	284
Total cold run time: 112230 ms
Total hot run time: 38027 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4225	4080	4068	4068
q2	321	228	230	228
q3	2952	2806	2854	2806
q4	1864	1555	1561	1555
q5	5230	5254	5305	5254
q6	197	116	118	116
q7	2278	1877	1835	1835
q8	3169	3326	3304	3304
q9	8598	8533	8614	8533
q10	3741	3678	3716	3678
q11	555	432	451	432
q12	738	549	539	539
q13	16915	2867	2869	2867
q14	284	251	263	251
q15	487	443	440	440
q16	458	415	409	409
q17	1743	1498	1487	1487
q18	7458	7238	7104	7104
q19	1629	1572	1521	1521
q20	1917	1702	1711	1702
q21	4964	4643	4764	4643
q22	529	432	450	432
Total cold run time: 70252 ms
Total hot run time: 53204 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181620 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f072c35369a365b105f1de996d43def0373a918, data reload: false

query1	909	366	359	359
query2	7502	2064	1992	1992
query3	6709	208	218	208
query4	30963	20691	20837	20691
query5	4966	514	402	402
query6	267	172	170	170
query7	4638	298	296	296
query8	228	171	179	171
query9	9890	2272	2244	2244
query10	464	248	267	248
query11	17351	14164	14347	14164
query12	137	89	86	86
query13	1617	410	406	406
query14	11424	10664	11433	10664
query15	327	189	196	189
query16	8220	273	259	259
query17	2043	582	545	545
query18	2096	297	280	280
query19	333	162	158	158
query20	91	90	89	89
query21	198	141	130	130
query22	4649	4485	4417	4417
query23	31929	31124	30981	30981
query24	10857	2859	2791	2791
query25	617	401	379	379
query26	1199	155	156	155
query27	2669	351	348	348
query28	7801	1868	1879	1868
query29	864	639	627	627
query30	303	152	150	150
query31	982	728	734	728
query32	94	59	60	59
query33	770	258	263	258
query34	1014	487	499	487
query35	828	600	599	599
query36	1005	859	859	859
query37	118	77	78	77
query38	3591	3474	3419	3419
query39	1428	1409	1383	1383
query40	207	114	116	114
query41	51	46	45	45
query42	105	96	97	96
query43	484	470	447	447
query44	1299	729	717	717
query45	275	269	254	254
query46	1071	689	704	689
query47	1661	1588	1568	1568
query48	452	367	365	365
query49	1105	334	335	334
query50	774	381	384	381
query51	6774	6604	6682	6604
query52	113	86	97	86
query53	342	282	282	282
query54	313	269	270	269
query55	82	86	76	76
query56	253	231	242	231
query57	1081	1027	1021	1021
query58	234	210	213	210
query59	2767	2658	2603	2603
query60	280	242	249	242
query61	129	114	112	112
query62	586	404	413	404
query63	307	282	295	282
query64	5486	3837	3795	3795
query65	3107	3001	3017	3001
query66	876	346	338	338
query67	14772	14330	14251	14251
query68	5389	523	515	515
query69	594	387	380	380
query70	1160	1205	1165	1165
query71	442	274	288	274
query72	6395	2688	2513	2513
query73	692	308	315	308
query74	7654	6583	6564	6564
query75	3737	3016	3003	3003
query76	3891	976	992	976
query77	611	277	270	270
query78	10370	9711	9673	9673
query79	8318	513	502	502
query80	1897	397	401	397
query81	550	210	211	210
query82	1627	210	196	196
query83	313	146	148	146
query84	292	76	79	76
query85	1644	332	313	313
query86	490	297	314	297
query87	3746	3586	3523	3523
query88	5162	2308	2314	2308
query89	520	369	373	369
query90	1985	185	185	185
query91	179	145	146	145
query92	64	49	48	48
query93	6449	503	493	493
query94	1201	186	186	186
query95	450	335	333	333
query96	612	277	276	276
query97	3050	2875	2864	2864
query98	230	215	211	211
query99	1075	779	748	748
Total cold run time: 306866 ms
Total hot run time: 181620 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 2f072c35369a365b105f1de996d43def0373a918 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

Copy link
Contributor

@zclllyybb zclllyybb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

case TYPE_DATETIMEV2: {
THROW_IF_ERROR(create_texpr_literal_node<TYPE_DATETIMEV2>(data, &node));
THROW_IF_ERROR(create_texpr_literal_node<TYPE_DATETIMEV2>(data, &node, precision, scale));
break;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

table1 probe:
2011-11-11 11:11:11.123456

table2 build:
2011-11-11 11:11:11.123455

@@ -97,11 +97,11 @@ TExprNode create_texpr_node_from(const void* data, const PrimitiveType& type, in
break;
}
case TYPE_DATEV2: {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODOs:

  1. verify datev1 and datetimev1 have precision ???
  2. table1 is datev2 and table2 is datev2 , generate runtime filter。
  3. table1 is datetimev2 and table2 is datetimev2 , generate runtime filter。

"""
sql """
CREATE TABLE `rftest_r` (
`k1_char` varchar(64)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rftest_r (string) ---> build hash table (datetimev2(0)) ---> min max (9999-12-31 23:59:59:999999) --->

consumer: liternal (9999-12-31 23:59:59:999999) (datetimev2(0))

@jacktengg
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.27% (8732/24757)
Line Coverage: 27.07% (71509/264172)
Region Coverage: 26.31% (37104/141007)
Branch Coverage: 23.21% (18975/81752)
Coverage Report: http://coverage.selectdb-in.cc/coverage/12a8a03db4adde9f0cc7ecf4bf886f0ad65346f4_12a8a03db4adde9f0cc7ecf4bf886f0ad65346f4/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 37983 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 12a8a03db4adde9f0cc7ecf4bf886f0ad65346f4, data reload: false

------ Round 1 ----------------------------------
q1	17645	4392	4106	4106
q2	2134	158	151	151
q3	10599	1179	1187	1179
q4	10222	812	795	795
q5	7493	2987	2964	2964
q6	206	121	126	121
q7	1031	586	560	560
q8	9339	2039	2027	2027
q9	7200	6536	6460	6460
q10	8410	3424	3545	3424
q11	433	224	215	215
q12	433	200	200	200
q13	17804	2862	2859	2859
q14	232	204	205	204
q15	503	465	466	465
q16	481	371	367	367
q17	959	551	580	551
q18	7213	6516	6432	6432
q19	4183	1457	1494	1457
q20	542	266	255	255
q21	3639	2916	2874	2874
q22	345	317	318	317
Total cold run time: 111046 ms
Total hot run time: 37983 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4136	4071	4071	4071
q2	337	239	236	236
q3	2995	2884	2828	2828
q4	1883	1575	1524	1524
q5	5252	5277	5288	5277
q6	196	118	119	118
q7	2255	1867	1821	1821
q8	3172	3351	3322	3322
q9	8631	8588	8645	8588
q10	3756	3715	3699	3699
q11	537	444	451	444
q12	706	572	569	569
q13	16927	2886	2867	2867
q14	289	269	247	247
q15	492	450	449	449
q16	478	415	425	415
q17	1773	1496	1518	1496
q18	7460	7103	7221	7103
q19	1611	1509	1518	1509
q20	1911	1733	1716	1716
q21	4816	4703	4583	4583
q22	569	472	458	458
Total cold run time: 70182 ms
Total hot run time: 53340 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181480 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 12a8a03db4adde9f0cc7ecf4bf886f0ad65346f4, data reload: false

query1	943	364	348	348
query2	7385	2002	1936	1936
query3	6715	211	212	211
query4	31081	20605	20712	20605
query5	4334	402	397	397
query6	263	172	175	172
query7	4633	293	291	291
query8	228	169	204	169
query9	9298	2328	2340	2328
query10	588	246	246	246
query11	17200	14112	14144	14112
query12	136	91	81	81
query13	1622	430	411	411
query14	11408	11046	11094	11046
query15	273	201	193	193
query16	8255	269	262	262
query17	1975	586	578	578
query18	2092	290	283	283
query19	355	152	159	152
query20	101	93	85	85
query21	202	133	129	129
query22	4673	4451	4435	4435
query23	31695	31415	31221	31221
query24	10743	2777	2798	2777
query25	633	382	411	382
query26	1186	160	158	158
query27	2415	354	370	354
query28	7333	1892	1922	1892
query29	900	655	630	630
query30	306	151	147	147
query31	960	726	727	726
query32	90	56	58	56
query33	771	256	262	256
query34	1012	486	506	486
query35	837	608	608	608
query36	1046	858	885	858
query37	122	78	77	77
query38	3537	3454	3492	3454
query39	1417	1384	1485	1384
query40	219	112	112	112
query41	51	49	48	48
query42	100	93	94	93
query43	492	446	438	438
query44	1144	732	725	725
query45	278	249	247	247
query46	1098	685	675	675
query47	1661	1590	1598	1590
query48	442	374	361	361
query49	1105	332	333	332
query50	768	375	376	375
query51	6689	6577	6556	6556
query52	113	90	89	89
query53	344	274	279	274
query54	320	248	259	248
query55	86	77	81	77
query56	258	232	230	230
query57	1084	998	1021	998
query58	252	211	213	211
query59	2710	2576	2501	2501
query60	267	258	264	258
query61	119	116	116	116
query62	606	404	386	386
query63	303	279	275	275
query64	5576	3892	3799	3799
query65	3010	3026	2991	2991
query66	873	339	342	339
query67	15133	14388	14188	14188
query68	6872	538	538	538
query69	620	381	377	377
query70	1269	1140	1131	1131
query71	505	287	277	277
query72	6422	2651	2485	2485
query73	731	323	323	323
query74	8105	6533	6571	6533
query75	4076	2886	2811	2811
query76	4769	903	841	841
query77	626	262	252	252
query78	10381	9641	9606	9606
query79	9433	541	517	517
query80	1676	397	391	391
query81	538	215	210	210
query82	1475	197	209	197
query83	211	144	149	144
query84	287	72	77	72
query85	1512	319	316	316
query86	483	313	281	281
query87	3776	3479	3501	3479
query88	4836	2385	2368	2368
query89	516	361	369	361
query90	1963	177	174	174
query91	175	141	134	134
query92	62	49	49	49
query93	6935	516	504	504
query94	1170	191	191	191
query95	446	340	343	340
query96	630	273	277	273
query97	3065	2873	2897	2873
query98	234	210	212	210
query99	1130	754	754	754
Total cold run time: 308793 ms
Total hot run time: 181480 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 12a8a03db4adde9f0cc7ecf4bf886f0ad65346f4 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.0 seconds inserted 10000000 Rows, about 500K ops/s

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.7-merged p0_w reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants