Skip to content

[fix](prepare) fix datetimev2 return err when binary_row_format #34662 #34807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nextdreamblue merged 24 commits into from
May 17, 2024
Merged

[fix](prepare) fix datetimev2 return err when binary_row_format #34662 #34807

nextdreamblue merged 24 commits into from
May 17, 2024

Conversation

@nextdreamblue
Copy link
Contributor

@nextdreamblue nextdreamblue commented May 14, 2024

Proposed changes

cherry-pick from #34662

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@nextdreamblue
[fix](prepare) fix datetimev2 return err when binary_row_format
90af8a2 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code
8113d3c 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code 3
76f1436 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code 4
b61e0d5 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code 5
7361e30 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix mysql datatimev2 scale
d2d1210 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code 7
49423a8 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code 8
3a0756c 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code 9
36fb295 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code 10
3ebf908 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code 11
f76cadc 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code 12
c85a3d7 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@doris-robot
Copy link

doris-robot commented May 14, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 14, 2024

run buildall

@nextdreamblue nextdreamblue changed the title Datetimev2 binary 2.0 [fix](prepare) fix datetimev2 return err when binary_row_format #34662 May 14, 2024
@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 14, 2024

run buildall

@nextdreamblue
fix code 2.0
a129772 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 14, 2024

run buildall

@nextdreamblue
fix code 2.0 1
13e1c99 
Signed-off-by: nextdreamblule <zxw520blue1@163.com>
@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 14, 2024

run buildall

@nextdreamblue
fix code 2.0 2
7a0daba 
Signed-off-by: nextdreamblule <zxw520blue1@163.com>
@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 14, 2024

run buildall

@nextdreamblue
fix code 2.0 4
cfd1931 
Signed-off-by: nextdreamblule <zxw520blue1@163.com>
@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 14, 2024

run buildall

@nextdreamblue
fix code 2.0 5
f3a9daf 
Signed-off-by: nextdreamblule <zxw520blue1@163.com>
@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 14, 2024

run buildall

@nextdreamblue
fix code 2.0 5
c7ccf14 
Signed-off-by: nextdreamblule <zxw520blue1@163.com>
@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 14, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 14, 2024

TeamCity be ut coverage result:
Function Coverage: 37.80% (8076/21366)
Line Coverage: 29.45% (65935/223876)
Region Coverage: 28.92% (33944/117366)
Branch Coverage: 24.78% (17417/70298)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c7ccf14147e87175cb925ea4255b0a1543c7398b_c7ccf14147e87175cb925ea4255b0a1543c7398b/report/index.html

@doris-robot
Copy link

doris-robot commented May 14, 2024

TPC-H: Total hot run time: 50439 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c7ccf14147e87175cb925ea4255b0a1543c7398b, data reload: false

------ Round 1 ----------------------------------
q1	17530	4404	4323	4323
q2	2036	154	156	154
q3	10260	1948	1930	1930
q4	10117	1259	1333	1259
q5	8517	3998	3981	3981
q6	239	124	126	124
q7	2017	1613	1606	1606
q8	9264	2729	2747	2729
q9	10601	10894	10763	10763
q10	8662	3534	3541	3534
q11	439	234	245	234
q12	483	296	300	296
q13	18367	3976	4041	3976
q14	347	327	321	321
q15	509	465	461	461
q16	689	570	588	570
q17	1158	993	969	969
q18	7274	6910	7017	6910
q19	1740	1595	1483	1483
q20	517	311	311	311
q21	4475	4188	4111	4111
q22	506	401	394	394
Total cold run time: 115747 ms
Total hot run time: 50439 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4341	4344	4314	4314
q2	322	228	227	227
q3	4156	4152	4134	4134
q4	2767	2757	2748	2748
q5	7249	7141	7128	7128
q6	238	124	116	116
q7	3272	2862	2891	2862
q8	4370	4498	4545	4498
q9	16908	16714	16797	16714
q10	4247	4277	4305	4277
q11	764	698	674	674
q12	1030	841	861	841
q13	6752	3774	3739	3739
q14	456	419	427	419
q15	507	464	453	453
q16	745	675	666	666
q17	3898	3950	3907	3907
q18	8719	8693	8833	8693
q19	1723	1742	1629	1629
q20	2353	2119	2077	2077
q21	8564	8550	8541	8541
q22	1082	947	908	908
Total cold run time: 84463 ms
Total hot run time: 79565 ms

@doris-robot
Copy link

doris-robot commented May 14, 2024

TPC-DS: Total hot run time: 203825 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c7ccf14147e87175cb925ea4255b0a1543c7398b, data reload: false

query1	932	402	373	373
query2	6521	3011	2556	2556
query3	6917	213	202	202
query4	21010	17963	17886	17886
query5	19732	6506	6488	6488
query6	278	214	223	214
query7	4155	299	305	299
query8	275	243	243	243
query9	3118	2718	2606	2606
query10	412	309	301	301
query11	11393	10651	10717	10651
query12	130	75	72	72
query13	5581	687	676	676
query14	17832	13520	13237	13237
query15	356	224	232	224
query16	6445	280	258	258
query17	1722	1447	863	863
query18	2333	415	396	396
query19	200	140	145	140
query20	82	77	79	77
query21	186	88	89	88
query22	5284	5070	4836	4836
query23	32390	31830	32026	31830
query24	7791	6522	6500	6500
query25	499	414	402	402
query26	1239	163	162	162
query27	2218	299	287	287
query28	6203	2383	2329	2329
query29	2875	2837	2743	2743
query30	241	165	164	164
query31	921	749	730	730
query32	63	62	56	56
query33	433	249	252	249
query34	870	481	481	481
query35	1114	928	935	928
query36	1394	1100	1393	1100
query37	172	57	59	57
query38	3100	2962	2951	2951
query39	1380	1324	1319	1319
query40	307	91	96	91
query41	38	36	34	34
query42	91	81	80	80
query43	742	731	626	626
query44	1104	715	721	715
query45	240	232	227	227
query46	1227	987	965	965
query47	1912	1727	1642	1642
query48	1048	723	711	711
query49	662	386	367	367
query50	871	625	611	611
query51	4706	4649	4567	4567
query52	91	77	88	77
query53	448	327	321	321
query54	2689	2452	2466	2452
query55	96	80	89	80
query56	220	217	215	215
query57	1238	1129	1233	1129
query58	208	198	185	185
query59	4322	4148	3779	3779
query60	203	192	218	192
query61	87	85	86	85
query62	850	416	466	416
query63	483	340	342	340
query64	2775	1530	1463	1463
query65	3646	3560	3536	3536
query66	805	377	366	366
query67	16785	17428	16028	16028
query68	5154	651	649	649
query69	533	347	368	347
query70	1742	1371	1311	1311
query71	373	315	318	315
query72	3978	3484	3474	3474
query73	721	327	312	312
query74	6380	5901	5897	5897
query75	4384	3792	3818	3792
query76	2961	1178	1206	1178
query77	443	247	255	247
query78	12440	12601	12522	12522
query79	10540	627	647	627
query80	1840	384	381	381
query81	526	233	229	229
query82	1047	105	98	98
query83	187	129	126	126
query84	263	69	67	67
query85	837	296	295	295
query86	352	292	296	292
query87	3236	3007	2980	2980
query88	5258	2360	2350	2350
query89	414	282	277	277
query90	1710	203	207	203
query91	168	148	136	136
query92	56	51	53	51
query93	5596	548	600	548
query94	765	198	195	195
query95	1100	1050	1063	1050
query96	656	331	331	331
query97	6362	6287	6380	6287
query98	184	173	164	164
query99	2774	836	820	820
Total cold run time: 308671 ms
Total hot run time: 203825 ms

@doris-robot
Copy link

doris-robot commented May 14, 2024

ClickBench: Total hot run time: 31.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c7ccf14147e87175cb925ea4255b0a1543c7398b, data reload: false

query1	0.02	0.03	0.02
query2	0.06	0.03	0.02
query3	0.25	0.04	0.05
query4	1.79	0.06	0.06
query5	0.53	0.52	0.53
query6	1.24	0.60	0.62
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.49	0.48
query10	0.54	0.53	0.52
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.62	0.61	0.61
query14	0.79	0.78	0.78
query15	0.79	0.77	0.77
query16	0.36	0.38	0.36
query17	1.00	1.04	1.05
query18	0.24	0.27	0.24
query19	1.93	1.81	1.85
query20	0.01	0.02	0.01
query21	15.47	0.54	0.55
query22	2.01	3.00	1.50
query23	17.20	0.98	1.08
query24	4.04	3.60	1.59
query25	0.39	0.09	0.06
query26	0.77	0.17	0.15
query27	0.04	0.04	0.03
query28	5.96	0.70	0.72
query29	12.63	2.35	2.29
query30	0.60	0.57	0.52
query31	2.80	0.37	0.36
query32	3.41	0.51	0.49
query33	3.07	3.08	3.06
query34	15.28	4.79	4.79
query35	4.85	4.82	4.84
query36	1.10	1.03	1.00
query37	0.05	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 100.99 s
Total hot run time: 31.21 s

@doris-robot
Copy link

doris-robot commented May 14, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c7ccf14147e87175cb925ea4255b0a1543c7398b with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.8 seconds inserted 10000000 Rows, about 458K ops/s

@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 14, 2024

run p0

@nextdreamblue nextdreamblue marked this pull request as ready for review May 15, 2024 02:22
@nextdreamblue
fix code 100
d8a714b 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code 101
365bec0 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
fix code 102
2fb5f67 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 16, 2024

run buildall

@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 16, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 16, 2024

TeamCity be ut coverage result:
Function Coverage: 37.84% (8086/21367)
Line Coverage: 29.48% (66016/223899)
Region Coverage: 28.97% (34000/117383)
Branch Coverage: 24.82% (17454/70312)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f6cf4bfb584b66756489d5c9aa76bb775231e720_f6cf4bfb584b66756489d5c9aa76bb775231e720/report/index.html

@nextdreamblue
fix code 103
2f43ad2 
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@nextdreamblue
Merge branch 'branch-2.0' into datetimev2_binary_2.0
ffcc05f
@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented May 17, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 17, 2024

TeamCity be ut coverage result:
Function Coverage: 37.80% (8078/21369)
Line Coverage: 29.46% (65956/223903)
Region Coverage: 28.93% (33959/117386)
Branch Coverage: 24.78% (17425/70314)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ffcc05fe76c564bd7316b20a92123791bf10d5df_ffcc05fe76c564bd7316b20a92123791bf10d5df/report/index.html

@nextdreamblue nextdreamblue requested a review from eldenmoon May 17, 2024 05:32
cambyzju
cambyzju approved these changes May 17, 2024
View reviewed changes
Copy link
Contributor

@cambyzju cambyzju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented May 17, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 17, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 17, 2024

PR approved by anyone and no changes requested.

eldenmoon
eldenmoon approved these changes May 17, 2024
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nextdreamblue nextdreamblue merged commit ea9dac5 into apache:branch-2.0 May 17, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@nextdreamblue
[fix](prepare) fix datetimev2 return err when binary_row_format apach…
ea0650d 
…e#34662 (apache#34807)

fix datetimev2 return err when binary_row_format. before pr, Backend return datetimev2 alwary by to_string.
fix datatimev2 return metadata loss scale.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cambyzju cambyzju cambyzju approved these changes

@eldenmoon eldenmoon eldenmoon approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. kind/test reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@nextdreamblue @doris-robot @cambyzju @eldenmoon