Skip to content

[Fix](nereids) fix condition function partition prune (#39298) #39331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Aug 16, 2024
Merged

[Fix](nereids) fix condition function partition prune (#39298) #39331

morrySnow merged 2 commits into from
Aug 16, 2024

Conversation

@feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Aug 14, 2024 •
edited
Loading

cherry-pick #39298 to branch-2.0

@feiniaofeiafei
[Fix](nereids) fix condition function partition prune (apache#39298)
93568d0 
Current partition prune logic cannot correctly deal with condition
function in predicates. For example, where if(substring('abc',1,2) <>
"case", a > 0, b > 0), if the first parameter is not fold by
FoldConstantRule, then our current logic will also intersect the
partition range of b with b>0. If one of the partitions is empty, the
partition will be trimmed. However, substring('abc',1,2) <> "case" is
true, the predicate b>0 is not useful and should not be involved in
partition trimming.
This pr remove the logic that make the result to "Boolean.False" when
considering the child ranges and finding an empty range.
@feiniaofeiafei feiniaofeiafei marked this pull request as draft August 14, 2024 03:17
@feiniaofeiafei feiniaofeiafei changed the base branch from branch-2.1 to branch-2.0 August 14, 2024 03:20
@feiniaofeiafei feiniaofeiafei marked this pull request as ready for review August 14, 2024 03:20
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Aug 14, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Aug 14, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Aug 14, 2024

TPC-H: Total hot run time: 49899 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 93568d0cb4a7078288822a7ce2dfab0b79302373, data reload: false

------ Round 1 ----------------------------------
q1	17624	4364	4364	4364
q2	2088	154	150	150
q3	10429	1884	1973	1884
q4	10301	1251	1319	1251
q5	8376	3966	3952	3952
q6	236	128	125	125
q7	2061	1590	1569	1569
q8	9302	2743	2701	2701
q9	10987	10620	10445	10445
q10	8622	3527	3499	3499
q11	427	252	248	248
q12	471	306	306	306
q13	18357	3965	4040	3965
q14	347	329	325	325
q15	518	469	452	452
q16	683	580	578	578
q17	1134	983	981	981
q18	7243	6766	6879	6766
q19	1696	1598	1540	1540
q20	545	315	313	313
q21	4463	4175	4081	4081
q22	505	404	410	404
Total cold run time: 116415 ms
Total hot run time: 49899 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4360	4296	4337	4296
q2	327	227	231	227
q3	4168	4176	4147	4147
q4	2752	2738	2747	2738
q5	7218	7105	7107	7105
q6	238	121	124	121
q7	3195	2825	2801	2801
q8	4338	4437	4491	4437
q9	17448	17298	16964	16964
q10	4233	4253	4285	4253
q11	760	687	688	687
q12	1028	844	868	844
q13	6958	3779	3700	3700
q14	443	414	417	414
q15	502	469	454	454
q16	739	689	670	670
q17	3745	3858	3855	3855
q18	8800	8810	8870	8810
q19	1711	1691	1625	1625
q20	2354	2175	2117	2117
q21	8423	8444	8384	8384
q22	1010	983	913	913
Total cold run time: 84750 ms
Total hot run time: 79562 ms

@doris-robot
Copy link

doris-robot commented Aug 14, 2024

TPC-DS: Total hot run time: 201500 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 93568d0cb4a7078288822a7ce2dfab0b79302373, data reload: false

query1	925	397	438	397
query2	6536	2242	2105	2105
query3	6917	206	201	201
query4	22069	18349	17944	17944
query5	19726	6571	6566	6566
query6	300	215	237	215
query7	4241	311	318	311
query8	232	263	245	245
query9	3159	2747	2627	2627
query10	414	311	309	309
query11	11379	10762	10620	10620
query12	127	74	75	74
query13	5596	687	677	677
query14	17940	13278	13578	13278
query15	363	219	227	219
query16	6449	291	268	268
query17	1700	1434	912	912
query18	2271	416	414	414
query19	222	147	154	147
query20	80	77	79	77
query21	188	101	95	95
query22	5251	5088	5083	5083
query23	32550	31951	31836	31836
query24	6958	6544	6528	6528
query25	535	444	426	426
query26	530	164	166	164
query27	1891	298	298	298
query28	6082	2299	2274	2274
query29	2949	2637	2830	2637
query30	249	171	170	170
query31	896	744	737	737
query32	69	59	59	59
query33	386	263	244	244
query34	866	479	480	479
query35	1162	923	974	923
query36	1327	1153	1189	1153
query37	88	61	65	61
query38	3086	2923	2939	2923
query39	1393	1326	1307	1307
query40	207	100	99	99
query41	40	37	36	36
query42	86	89	86	86
query43	660	664	608	608
query44	1127	716	704	704
query45	243	233	230	230
query46	1223	943	962	943
query47	1783	1855	1663	1663
query48	1002	677	697	677
query49	618	391	389	389
query50	876	613	626	613
query51	4825	4658	4708	4658
query52	87	83	86	83
query53	445	328	317	317
query54	2652	2480	2448	2448
query55	83	86	85	85
query56	241	215	209	209
query57	1288	1069	1194	1069
query58	219	203	213	203
query59	3563	3304	3254	3254
query60	218	202	206	202
query61	100	101	95	95
query62	807	454	470	454
query63	490	342	341	341
query64	2381	1482	1377	1377
query65	3648	3585	3530	3530
query66	798	376	381	376
query67	15964	15448	14976	14976
query68	9973	653	661	653
query69	569	356	355	355
query70	1829	1671	1500	1500
query71	428	304	303	303
query72	6560	3470	3477	3470
query73	740	320	318	318
query74	6265	5764	5830	5764
query75	5269	3736	3700	3700
query76	6252	1181	1098	1098
query77	1062	257	246	246
query78	12943	11735	12767	11735
query79	6984	654	644	644
query80	1100	397	412	397
query81	483	236	228	228
query82	688	96	99	96
query83	176	131	129	129
query84	264	74	71	71
query85	896	323	320	320
query86	336	304	291	291
query87	3245	2981	2995	2981
query88	4216	2316	2301	2301
query89	357	288	276	276
query90	1953	212	214	212
query91	166	124	139	124
query92	60	50	53	50
query93	4483	593	578	578
query94	692	210	203	203
query95	1118	1061	1059	1059
query96	648	323	334	323
query97	6570	6365	6464	6365
query98	196	172	172	172
query99	3011	870	810	810
Total cold run time: 312308 ms
Total hot run time: 201500 ms

@doris-robot
Copy link

doris-robot commented Aug 14, 2024

ClickBench: Total hot run time: 29.98 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 93568d0cb4a7078288822a7ce2dfab0b79302373, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.05	0.04
query4	1.77	0.06	0.06
query5	0.54	0.53	0.51
query6	1.23	0.62	0.67
query7	0.01	0.01	0.02
query8	0.03	0.03	0.02
query9	0.54	0.49	0.49
query10	0.54	0.54	0.54
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.62
query14	0.80	0.79	0.80
query15	0.77	0.76	0.74
query16	0.37	0.36	0.37
query17	1.02	1.00	1.01
query18	0.23	0.25	0.24
query19	1.85	1.83	1.81
query20	0.01	0.01	0.02
query21	15.46	0.56	0.56
query22	2.04	2.04	1.10
query23	17.00	0.96	0.96
query24	7.17	0.72	0.82
query25	0.37	0.06	0.07
query26	0.77	0.15	0.15
query27	0.04	0.04	0.04
query28	6.63	0.71	0.70
query29	12.66	2.37	2.27
query30	0.59	0.51	0.51
query31	2.81	0.39	0.38
query32	3.37	0.49	0.49
query33	3.05	3.08	3.04
query34	15.25	4.80	4.80
query35	4.89	4.87	4.86
query36	1.08	1.02	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.03	0.01	0.01
Total cold run time: 104.49 s
Total hot run time: 29.98 s

@doris-robot
Copy link

doris-robot commented Aug 14, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 93568d0cb4a7078288822a7ce2dfab0b79302373 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 16, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 16, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 16, 2024

PR approved by anyone and no changes requested.

@morrySnow
Copy link
Contributor

morrySnow commented Aug 16, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 16, 2024

TPC-H: Total hot run time: 50447 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 11ef22ef399a7c1da6993bb208c5776d7b7d8ab7, data reload: false

------ Round 1 ----------------------------------
q1	17799	4493	4458	4458
q2	2086	204	189	189
q3	10173	1900	1979	1900
q4	10156	1285	1374	1285
q5	8472	3974	3984	3974
q6	275	144	143	143
q7	2097	1646	1650	1646
q8	9578	2788	2752	2752
q9	13710	10432	10208	10208
q10	8691	3574	3565	3565
q11	445	277	286	277
q12	498	329	333	329
q13	18389	3992	4075	3992
q14	386	350	359	350
q15	559	501	509	501
q16	692	604	600	600
q17	1153	963	926	926
q18	7389	6918	6832	6832
q19	1765	1612	1603	1603
q20	545	345	321	321
q21	4461	4234	4159	4159
q22	526	442	437	437
Total cold run time: 119845 ms
Total hot run time: 50447 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4355	4311	4360	4311
q2	364	268	269	268
q3	4212	4138	4181	4138
q4	2791	2780	2755	2755
q5	7250	7171	7156	7156
q6	256	139	137	137
q7	3346	2912	2943	2912
q8	4426	4500	4554	4500
q9	16936	16864	16867	16864
q10	4275	4271	4291	4271
q11	793	708	700	700
q12	1050	866	879	866
q13	6805	3775	3730	3730
q14	487	442	453	442
q15	544	502	508	502
q16	768	702	725	702
q17	3837	3938	3872	3872
q18	8897	8684	8711	8684
q19	1783	1722	1736	1722
q20	2404	2175	2113	2113
q21	8557	8438	8538	8438
q22	1070	939	964	939
Total cold run time: 85206 ms
Total hot run time: 80022 ms

@doris-robot
Copy link

doris-robot commented Aug 16, 2024

TPC-DS: Total hot run time: 209880 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 11ef22ef399a7c1da6993bb208c5776d7b7d8ab7, data reload: false

query1	1013	422	415	415
query2	6795	2420	2123	2123
query3	6946	229	225	225
query4	20440	18310	18271	18271
query5	19926	6701	6730	6701
query6	328	265	275	265
query7	4182	332	342	332
query8	482	462	452	452
query9	3250	2781	2736	2736
query10	485	369	376	369
query11	11538	10919	10901	10901
query12	156	107	115	107
query13	5678	740	734	734
query14	18086	14149	14154	14149
query15	391	270	249	249
query16	6513	333	305	305
query17	1559	1514	951	951
query18	2328	494	485	485
query19	250	186	186	186
query20	112	114	117	114
query21	246	147	147	147
query22	5325	5113	5065	5065
query23	32585	32029	32062	32029
query24	7081	6611	6570	6570
query25	582	479	474	474
query26	558	194	191	191
query27	1839	340	332	332
query28	5988	2445	2386	2386
query29	3011	2899	2877	2877
query30	287	209	207	207
query31	1008	812	838	812
query32	99	92	90	90
query33	474	330	346	330
query34	906	548	539	539
query35	1157	1011	1000	1000
query36	1436	1192	1353	1192
query37	121	89	93	89
query38	3146	2974	2938	2938
query39	1482	1444	1461	1444
query40	252	147	151	147
query41	157	160	156	156
query42	114	120	109	109
query43	818	700	615	615
query44	1207	788	794	788
query45	288	271	269	269
query46	1284	1010	1019	1010
query47	1860	1905	1881	1881
query48	1058	742	737	737
query49	835	586	598	586
query50	946	675	650	650
query51	4836	4742	4695	4695
query52	108	104	113	104
query53	506	383	378	378
query54	2737	2552	2566	2552
query55	98	106	97	97
query56	330	300	287	287
query57	1296	1226	1081	1081
query58	301	310	313	310
query59	3580	3395	3297	3297
query60	298	299	289	289
query61	162	158	162	158
query62	889	603	537	537
query63	540	396	399	396
query64	2745	1738	1497	1497
query65	3667	3634	3620	3620
query66	1226	818	815	815
query67	16054	15615	15228	15228
query68	8529	721	713	713
query69	625	435	413	413
query70	1584	1589	1486	1486
query71	458	358	373	358
query72	6588	3546	3561	3546
query73	791	366	366	366
query74	6310	5988	5938	5938
query75	4621	3848	3855	3848
query76	4711	1207	1265	1207
query77	755	420	430	420
query78	12742	11506	12380	11506
query79	10458	703	694	694
query80	1457	554	552	552
query81	547	287	276	276
query82	1632	124	125	124
query83	255	225	219	219
query84	291	98	101	98
query85	1064	410	412	410
query86	379	365	330	330
query87	3260	3037	3121	3037
query88	5437	2545	2555	2545
query89	524	342	336	336
query90	1928	255	260	255
query91	195	168	164	164
query92	93	87	84	84
query93	6313	633	616	616
query94	826	253	249	249
query95	1155	1120	1109	1109
query96	669	351	346	346
query97	6576	6400	6392	6392
query98	229	214	205	205
query99	2935	967	1066	967
Total cold run time: 319918 ms
Total hot run time: 209880 ms

@doris-robot
Copy link

doris-robot commented Aug 16, 2024

ClickBench: Total hot run time: 31.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 11ef22ef399a7c1da6993bb208c5776d7b7d8ab7, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.03
query3	0.25	0.05	0.05
query4	1.79	0.07	0.06
query5	0.54	0.52	0.52
query6	1.23	0.61	0.64
query7	0.01	0.01	0.02
query8	0.04	0.03	0.03
query9	0.54	0.48	0.48
query10	0.56	0.55	0.56
query11	0.13	0.09	0.10
query12	0.12	0.10	0.11
query13	0.63	0.63	0.61
query14	0.78	0.79	0.79
query15	0.78	0.76	0.75
query16	0.38	0.39	0.36
query17	1.06	1.02	1.02
query18	0.24	0.25	0.23
query19	1.92	1.87	1.78
query20	0.02	0.02	0.01
query21	15.47	0.54	0.54
query22	2.15	2.36	2.11
query23	17.48	0.96	0.99
query24	7.55	1.44	0.72
query25	0.35	0.07	0.07
query26	0.90	0.16	0.17
query27	0.04	0.05	0.05
query28	5.45	0.80	0.74
query29	12.66	2.37	2.32
query30	0.79	0.71	0.73
query31	2.82	0.40	0.38
query32	3.36	0.51	0.51
query33	3.09	3.05	3.08
query34	15.27	4.81	4.78
query35	4.85	4.85	4.85
query36	1.06	1.03	1.03
query37	0.07	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.17	0.17
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 104.91 s
Total hot run time: 31.48 s

@doris-robot
Copy link

doris-robot commented Aug 16, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 11ef22ef399a7c1da6993bb208c5776d7b7d8ab7 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@morrySnow morrySnow merged commit 787cf16 into apache:branch-2.0 Aug 16, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@feiniaofeiafei @Baymine
[Fix](nereids) fix condition function partition prune (apache#39298) (a…
5818ea8 
…pache#39331)

cherry-pick from master apache#39298
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. area/nereids kind/test reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@feiniaofeiafei @doris-robot @morrySnow