Skip to content

Conversation

@GoGoWen
Copy link
Contributor

@GoGoWen GoGoWen commented Jan 19, 2024

Proposed changes

1, have iceberg table with partitioned by date type column "day"

2 execute select * from test_table where day = "2022-01-01" will get nothing while the data exist in iceberg table, just we transfer the DateLiteral("2022-01-01") to datetime, cause we can not get the split correctly.

bp #30162

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions github-actions bot added the area/planner Issues or PRs related to the query planner label Jan 19, 2024
@GoGoWen
Copy link
Contributor Author

GoGoWen commented Jan 19, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49979 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 50238b2f89518e904fb6acd7b216b8899840814a, data reload: false

------ Round 1 ----------------------------------
q1	18031	4470	4307	4307
q2	2085	157	140	140
q3	10280	1870	1927	1870
q4	10096	1268	1325	1268
q5	8476	3909	3974	3909
q6	231	125	124	124
q7	2078	1601	1600	1600
q8	9327	2728	2722	2722
q9	10764	10585	10571	10571
q10	8653	3509	3510	3509
q11	412	245	246	245
q12	464	300	298	298
q13	18367	3950	4054	3950
q14	350	324	322	322
q15	509	456	456	456
q16	698	594	594	594
q17	1133	960	1000	960
q18	7285	6811	6925	6811
q19	1752	1538	1537	1537
q20	522	313	299	299
q21	4437	4126	4086	4086
q22	499	401	412	401
Total cold run time: 116449 ms
Total hot run time: 49979 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4310	4290	4272	4272
q2	323	228	228	228
q3	4181	4193	4138	4138
q4	2734	2737	2747	2737
q5	7346	7231	7224	7224
q6	234	115	116	115
q7	3228	2893	2891	2891
q8	4397	4523	4485	4485
q9	17136	16981	16960	16960
q10	4208	4255	4241	4241
q11	759	696	686	686
q12	1025	852	833	833
q13	6827	3761	3763	3761
q14	460	431	419	419
q15	497	447	443	443
q16	767	720	701	701
q17	3894	3825	3893	3825
q18	8746	8848	9003	8848
q19	1721	1712	1672	1672
q20	2384	2142	2085	2085
q21	8708	8498	8492	8492
q22	1026	996	955	955
Total cold run time: 84911 ms
Total hot run time: 80011 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 239935 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 50238b2f89518e904fb6acd7b216b8899840814a, data reload: false

query1	914	390	392	390
query2	6539	2147	2004	2004
query3	6928	206	196	196
query4	21516	18006	17938	17938
query5	19724	6213	6240	6213
query6	279	210	230	210
query7	4139	288	295	288
query8	249	228	243	228
query9	3082	2682	2627	2627
query10	416	299	309	299
query11	11404	10790	10783	10783
query12	122	80	74	74
query13	5652	637	628	628
query14	17402	13236	13377	13236
query15	361	244	237	237
query16	6453	264	259	259
query17	3195	874	873	873
query18	2265	406	398	398
query19	207	151	153	151
query20	77	78	75	75
query21	187	99	101	99
query22	5235	5180	4981	4981
query23	32641	32027	31884	31884
query24	7004	6498	6499	6498
query25	530	423	426	423
query26	615	160	164	160
query27	2026	294	294	294
query28	6087	2237	2216	2216
query29	2886	2735	2866	2735
query30	246	160	163	160
query31	909	721	722	721
query32	67	58	57	57
query33	385	255	254	254
query34	841	475	475	475
query35	1132	959	912	912
query36	1553	1588	1659	1588
query37	85	62	58	58
query38	3092	2973	2956	2956
query39	1364	1310	1338	1310
query40	198	99	95	95
query41	34	31	30	30
query42	87	82	80	80
query43	691	551	616	551
query44	1134	714	711	711
query45	235	230	229	229
query46	1211	954	978	954
query47	1842	1644	1773	1644
query48	970	682	673	673
query49	618	377	375	375
query50	858	595	607	595
query51	5733	5580	5521	5521
query52	89	86	77	77
query53	437	326	318	318
query54	2422	2204	2254	2204
query55	81	77	74	74
query56	203	211	186	186
query57	1124	1143	1135	1135
query58	215	213	199	199
query59	3467	3058	3348	3058
query60	199	205	204	204
query61	85	81	85	81
query62	811	502	483	483
query63	468	335	333	333
query64	2390	1292	1296	1292
query65	3690	3617	3568	3568
query66	878	371	355	355
query67	18670	17609	16083	16083
query68	6797	621	650	621
query69	549	353	339	339
query70	1826	1799	1659	1659
query71	358	312	314	312
query72	4410	3373	3356	3356
query73	714	313	314	313
query74	6333	5868	5908	5868
query75	4257	3511	3574	3511
query76	4412	1124	1154	1124
query77	534	253	242	242
query78	31740	54399	48980	48980
query79	15697	641	647	641
query80	4564	391	383	383
query81	573	225	231	225
query82	1072	97	94	94
query83	429	137	132	132
query84	259	68	66	66
query85	2336	281	274	274
query86	448	387	369	369
query87	3240	2991	3042	2991
query88	6981	2300	2293	2293
query89	447	293	287	287
query90	2506	211	213	211
query91	153	123	119	119
query92	65	51	49	49
query93	6357	568	556	556
query94	1670	205	208	205
query95	1098	1084	1069	1069
query96	644	326	328	326
query97	6546	6301	6357	6301
query98	188	167	167	167
query99	3674	856	924	856
Total cold run time: 347526 ms
Total hot run time: 239935 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 50238b2f89518e904fb6acd7b216b8899840814a, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.24	0.05	0.05
query4	1.84	0.07	0.08
query5	0.54	0.53	0.52
query6	1.23	0.62	0.61
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.47	0.49
query10	0.53	0.53	0.54
query11	0.10	0.08	0.09
query12	0.11	0.09	0.09
query13	0.61	0.62	0.61
query14	0.80	0.79	0.79
query15	0.77	0.76	0.75
query16	0.38	0.37	0.35
query17	0.96	0.98	1.00
query18	0.22	0.26	0.24
query19	1.85	1.83	1.87
query20	0.01	0.01	0.01
query21	15.52	0.57	0.57
query22	2.08	1.98	1.83
query23	17.59	1.04	0.93
query24	4.48	3.67	1.33
query25	2.39	0.16	0.14
query26	0.15	0.13	0.14
query27	0.15	0.16	0.15
query28	5.23	0.75	0.71
query29	12.64	2.22	2.31
query30	0.61	0.55	0.60
query31	2.82	0.39	0.38
query32	3.42	0.50	0.49
query33	3.05	3.07	3.12
query34	15.27	4.79	4.80
query35	4.86	4.81	4.82
query36	1.05	1.01	1.01
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.01	0.02	0.02
query40	0.15	0.13	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 102.51 s
Total hot run time: 31.4 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 50238b2f89518e904fb6acd7b216b8899840814a with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50264 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4eb8e1a88a584da5b84176d106191065bfbcf24e, data reload: false

------ Round 1 ----------------------------------
q1	17551	4317	4311	4311
q2	2035	155	139	139
q3	10280	1880	1958	1880
q4	10091	1261	1313	1261
q5	8423	4024	4034	4024
q6	226	123	125	123
q7	2050	1598	1602	1598
q8	9301	2729	2716	2716
q9	11152	11520	10816	10816
q10	8668	3567	3506	3506
q11	410	247	246	246
q12	461	298	296	296
q13	18359	3975	4034	3975
q14	347	341	310	310
q15	505	461	456	456
q16	701	595	598	595
q17	1146	941	935	935
q18	7372	6744	6877	6744
q19	1679	1588	1519	1519
q20	511	323	324	323
q21	4412	4094	4110	4094
q22	517	410	397	397
Total cold run time: 116197 ms
Total hot run time: 50264 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4334	4320	4278	4278
q2	323	219	227	219
q3	4199	4149	4172	4149
q4	2764	2742	2721	2721
q5	7359	7238	7259	7238
q6	233	118	117	117
q7	3224	2844	2781	2781
q8	4365	4473	4450	4450
q9	17565	17432	17319	17319
q10	4209	4256	4290	4256
q11	770	670	710	670
q12	1027	863	836	836
q13	4325	3732	3797	3732
q14	449	437	449	437
q15	496	447	461	447
q16	754	710	719	710
q17	3849	3869	3837	3837
q18	8736	8788	8800	8788
q19	1707	1704	1650	1650
q20	2355	2128	2080	2080
q21	8528	8461	8454	8454
q22	1016	905	907	905
Total cold run time: 82587 ms
Total hot run time: 80074 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 235630 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4eb8e1a88a584da5b84176d106191065bfbcf24e, data reload: false

query1	933	388	388	388
query2	6513	2272	2083	2083
query3	6918	210	205	205
query4	20656	18022	18185	18022
query5	19726	6300	6253	6253
query6	279	219	241	219
query7	4154	296	310	296
query8	262	251	216	216
query9	3078	2657	2576	2576
query10	413	297	299	297
query11	11407	10773	10697	10697
query12	118	70	76	70
query13	5635	621	626	621
query14	17278	13386	13377	13377
query15	358	232	231	231
query16	6458	264	269	264
query17	3177	872	866	866
query18	2287	402	405	402
query19	211	147	148	147
query20	78	74	71	71
query21	189	95	93	93
query22	5313	5211	5174	5174
query23	32527	31793	31921	31793
query24	7048	6484	6514	6484
query25	507	429	409	409
query26	523	156	155	155
query27	1907	300	292	292
query28	6092	2223	2196	2196
query29	2977	2705	2707	2705
query30	238	158	158	158
query31	889	718	730	718
query32	66	59	58	58
query33	388	243	239	239
query34	853	461	474	461
query35	1089	900	905	900
query36	1881	1503	1485	1485
query37	90	62	59	59
query38	3054	2952	2913	2913
query39	1367	1324	1318	1318
query40	207	97	92	92
query41	33	32	32	32
query42	82	83	81	81
query43	675	597	534	534
query44	1112	719	710	710
query45	240	232	224	224
query46	1231	952	982	952
query47	1827	1670	1712	1670
query48	965	661	649	649
query49	607	368	376	368
query50	858	572	600	572
query51	5621	5529	5509	5509
query52	89	72	81	72
query53	435	324	313	313
query54	2413	2258	2242	2242
query55	80	75	77	75
query56	219	211	200	200
query57	1181	1192	1053	1053
query58	209	216	193	193
query59	3615	3342	3236	3236
query60	202	195	196	195
query61	90	82	87	82
query62	784	528	504	504
query63	475	328	330	328
query64	2036	1340	1296	1296
query65	3672	3642	3616	3616
query66	796	391	367	367
query67	15843	15494	14877	14877
query68	10888	652	649	649
query69	581	338	351	338
query70	2299	1968	1664	1664
query71	395	303	293	293
query72	4421	3288	3294	3288
query73	726	325	314	314
query74	6364	5922	5874	5874
query75	5058	3507	3540	3507
query76	7023	1184	1218	1184
query77	1175	241	237	237
query78	32245	52595	45868	45868
query79	14514	619	623	619
query80	4577	384	377	377
query81	562	226	228	226
query82	715	99	93	93
query83	365	131	128	128
query84	261	69	65	65
query85	2504	276	269	269
query86	480	378	383	378
query87	3324	2952	2978	2952
query88	6324	2326	2295	2295
query89	373	303	269	269
query90	2478	207	208	207
query91	152	123	113	113
query92	67	49	47	47
query93	4022	551	532	532
query94	1793	200	205	200
query95	1096	1063	1056	1056
query96	656	328	319	319
query97	6511	6282	6331	6282
query98	186	168	171	168
query99	3894	866	975	866
Total cold run time: 348563 ms
Total hot run time: 235630 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.26 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4eb8e1a88a584da5b84176d106191065bfbcf24e, data reload: false

query1	0.03	0.02	0.01
query2	0.06	0.02	0.02
query3	0.24	0.04	0.05
query4	1.82	0.06	0.07
query5	0.53	0.53	0.52
query6	1.23	0.61	0.63
query7	0.01	0.01	0.00
query8	0.03	0.02	0.02
query9	0.52	0.47	0.47
query10	0.54	0.53	0.54
query11	0.11	0.08	0.08
query12	0.12	0.10	0.10
query13	0.61	0.63	0.62
query14	0.77	0.78	0.78
query15	0.78	0.77	0.76
query16	0.35	0.40	0.37
query17	1.01	1.02	0.99
query18	0.24	0.27	0.24
query19	1.95	1.84	1.80
query20	0.01	0.01	0.01
query21	15.49	0.56	0.55
query22	1.80	1.79	1.61
query23	17.38	1.03	0.89
query24	8.73	1.49	0.58
query25	1.59	0.11	0.11
query26	0.39	0.14	0.13
query27	0.10	0.10	0.10
query28	5.33	0.74	0.71
query29	12.62	2.21	2.37
query30	0.55	0.51	0.52
query31	2.80	0.38	0.39
query32	3.42	0.49	0.50
query33	3.09	3.06	3.05
query34	15.27	4.82	4.78
query35	4.88	4.85	4.86
query36	1.04	1.03	1.02
query37	0.05	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.82 s
Total hot run time: 30.26 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4eb8e1a88a584da5b84176d106191065bfbcf24e with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@morningman morningman merged commit 0f3700d into apache:branch-2.0 Feb 1, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/planner Issues or PRs related to the query planner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants