Skip to content

[fix](iceberg) fix error when query iceberg table using date cast predicate #32211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
lide-reed merged 1 commit into from
Mar 14, 2024
Merged

[fix](iceberg) fix error when query iceberg table using date cast predicate #32211

lide-reed merged 1 commit into from
Mar 14, 2024

Conversation

@Yulei-Yang
Copy link
Contributor

@Yulei-Yang Yulei-Yang commented Mar 14, 2024

bp #32194

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Mar 14, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@github-actions github-actions bot added the area/planner Issues or PRs related to the query planner label Mar 14, 2024
@lide-reed lide-reed self-requested a review March 14, 2024 02:10
lide-reed
lide-reed approved these changes Mar 14, 2024
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 14, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 14, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 14, 2024

PR approved by anyone and no changes requested.

cambyzju
cambyzju approved these changes Mar 14, 2024
View reviewed changes
Copy link
Contributor

@cambyzju cambyzju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Yulei-Yang
Copy link
Contributor Author

Yulei-Yang commented Mar 14, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 14, 2024

TPC-H: Total hot run time: 50172 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d4b20e348b9b629b4fd548e59f37bb7384338112, data reload: false

------ Round 1 ----------------------------------
q1	17582	4439	4380	4380
q2	2050	153	144	144
q3	10369	1915	1963	1915
q4	10341	1249	1355	1249
q5	8376	4007	4021	4007
q6	232	121	122	121
q7	2029	1624	1602	1602
q8	9564	2753	2754	2753
q9	14048	10565	10461	10461
q10	8646	3547	3536	3536
q11	413	252	243	243
q12	463	297	300	297
q13	18329	3986	4042	3986
q14	340	323	327	323
q15	509	471	461	461
q16	696	590	603	590
q17	1138	948	934	934
q18	7361	6872	6988	6872
q19	1682	1562	1524	1524
q20	513	309	289	289
q21	4471	4184	4078	4078
q22	508	407	413	407
Total cold run time: 119660 ms
Total hot run time: 50172 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4310	4344	4316	4316
q2	324	219	224	219
q3	4189	4153	4173	4153
q4	2777	2759	2782	2759
q5	7287	7260	7250	7250
q6	238	119	116	116
q7	3233	2834	2945	2834
q8	4381	4530	4512	4512
q9	17074	17060	16989	16989
q10	4243	4301	4293	4293
q11	741	678	704	678
q12	1022	857	831	831
q13	6711	3771	3753	3753
q14	453	415	424	415
q15	504	454	448	448
q16	752	706	708	706
q17	3868	3910	3830	3830
q18	8762	8894	8829	8829
q19	1750	1713	1676	1676
q20	2373	2154	2119	2119
q21	8594	8625	8605	8605
q22	1006	975	957	957
Total cold run time: 84592 ms
Total hot run time: 80288 ms

@doris-robot
Copy link

doris-robot commented Mar 14, 2024

TPC-DS: Total hot run time: 201617 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d4b20e348b9b629b4fd548e59f37bb7384338112, data reload: false

query1	921	399	382	382
query2	6542	2342	2127	2127
query3	6914	201	195	195
query4	20715	17886	17830	17830
query5	19701	6524	6474	6474
query6	301	214	243	214
query7	4162	293	297	293
query8	247	264	229	229
query9	3122	2676	2602	2602
query10	410	294	305	294
query11	11346	10678	10714	10678
query12	122	73	70	70
query13	5577	627	638	627
query14	18157	14173	13530	13530
query15	372	226	241	226
query16	6453	269	265	265
query17	1739	1427	862	862
query18	2353	400	393	393
query19	194	141	151	141
query20	78	75	80	75
query21	184	95	96	95
query22	5257	5070	5047	5047
query23	32639	31980	31840	31840
query24	7013	6628	6509	6509
query25	536	435	419	419
query26	521	161	155	155
query27	1887	295	288	288
query28	6031	2215	2190	2190
query29	2885	2726	2827	2726
query30	240	160	162	160
query31	893	728	742	728
query32	67	57	59	57
query33	396	258	243	243
query34	853	448	466	448
query35	1120	932	902	902
query36	1738	1645	1483	1483
query37	84	68	60	60
query38	3084	2920	2907	2907
query39	1388	1321	1328	1321
query40	201	90	95	90
query41	36	31	32	31
query42	84	80	83	80
query43	686	586	571	571
query44	1109	706	702	702
query45	240	231	229	229
query46	1251	970	976	970
query47	1733	1752	1788	1752
query48	976	668	640	640
query49	608	376	359	359
query50	884	637	582	582
query51	4783	4614	4566	4566
query52	102	73	75	73
query53	440	314	316	314
query54	2659	2502	2497	2497
query55	77	89	72	72
query56	210	209	206	206
query57	1135	1087	1046	1046
query58	204	208	193	193
query59	3704	3437	3028	3028
query60	200	192	199	192
query61	81	80	84	80
query62	861	529	531	529
query63	473	327	337	327
query64	2370	1486	1337	1337
query65	3633	3580	3594	3580
query66	827	374	358	358
query67	15958	16099	15255	15255
query68	7115	628	658	628
query69	579	341	349	341
query70	1969	1824	1731	1731
query71	391	305	307	305
query72	6412	3463	3413	3413
query73	720	324	319	319
query74	6282	5905	5722	5722
query75	4584	3725	3737	3725
query76	4396	1143	1182	1143
query77	530	242	243	242
query78	12505	11514	11570	11514
query79	6868	640	632	632
query80	2538	397	384	384
query81	533	234	227	227
query82	1551	100	98	98
query83	183	129	131	129
query84	257	67	67	67
query85	1398	274	274	274
query86	407	384	430	384
query87	3247	3018	3006	3006
query88	4995	2322	2312	2312
query89	359	295	322	295
query90	1686	204	202	202
query91	155	111	118	111
query92	54	49	52	49
query93	2518	564	598	564
query94	884	205	200	200
query95	1145	1090	1052	1052
query96	630	326	323	323
query97	6491	6427	6405	6405
query98	187	186	171	171
query99	3004	1014	905	905
Total cold run time: 306340 ms
Total hot run time: 201617 ms

@doris-robot
Copy link

doris-robot commented Mar 14, 2024

ClickBench: Total hot run time: 31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d4b20e348b9b629b4fd548e59f37bb7384338112, data reload: false

query1	0.03	0.02	0.01
query2	0.06	0.02	0.02
query3	0.24	0.05	0.06
query4	1.82	0.11	0.08
query5	0.53	0.53	0.52
query6	1.23	0.63	0.62
query7	0.02	0.01	0.00
query8	0.04	0.03	0.02
query9	0.53	0.48	0.48
query10	0.53	0.53	0.54
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.61	0.62	0.60
query14	0.79	0.78	0.79
query15	0.79	0.76	0.76
query16	0.36	0.35	0.36
query17	1.00	1.00	1.02
query18	0.23	0.25	0.26
query19	1.91	1.87	1.84
query20	0.01	0.00	0.01
query21	15.48	0.55	0.56
query22	1.96	2.26	1.52
query23	17.26	0.98	0.96
query24	6.66	2.18	1.36
query25	0.36	0.11	0.05
query26	0.84	0.15	0.15
query27	0.04	0.04	0.03
query28	5.16	0.75	0.71
query29	12.71	2.27	2.25
query30	0.62	0.57	0.51
query31	2.80	0.38	0.39
query32	3.41	0.49	0.50
query33	3.08	3.05	3.08
query34	15.25	4.82	4.82
query35	4.87	4.86	4.83
query36	1.06	1.01	1.01
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 102.9 s
Total hot run time: 31 s

@doris-robot
Copy link

doris-robot commented Mar 14, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d4b20e348b9b629b4fd548e59f37bb7384338112 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.5 seconds inserted 10000000 Rows, about 487K ops/s

@lide-reed lide-reed merged commit 1e9be3d into apache:branch-2.0 Mar 14, 2024
@Yulei-Yang Yulei-Yang deleted the fix_iceberg_date_predicate_for2.0 branch May 20, 2024 09:26
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@Yulei-Yang
[fix](iceberg) fix error when query iceberg table using date cast pre…
d3efc19 
…dicate (apache#32211)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lide-reed lide-reed lide-reed approved these changes

@cambyzju cambyzju cambyzju approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. area/planner Issues or PRs related to the query planner reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Yulei-Yang @doris-robot @lide-reed @cambyzju