Skip to content

[enhancement](nereids) Support eliminate outer join by match expression #38537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
924060929 merged 8 commits into from
Aug 6, 2024
Merged

[enhancement](nereids) Support eliminate outer join by match expression #38537

924060929 merged 8 commits into from
Aug 6, 2024

Conversation

@924060929
Copy link
Contributor

@924060929 924060929 commented Jul 30, 2024 •
edited
Loading

Proposed changes

  1. enable run match expression outer of filter plan, e.g join conjunct
  2. support eliminate outer join by match expression, if any arguments of match expression is null literal

@doris-robot
Copy link

doris-robot commented Jul 30, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@924060929
Copy link
Contributor Author

924060929 commented Jul 30, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 30, 2024

TPC-H: Total hot run time: 41632 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 19a35422b4a6e27469cc956a3359b9c24f950217, data reload: false

------ Round 1 ----------------------------------
q1	17603	4053	4028	4028
q2	2026	206	206	206
q3	10598	1252	1351	1252
q4	10609	814	951	814
q5	7900	3015	3058	3015
q6	225	138	142	138
q7	1074	619	620	619
q8	9614	1942	1949	1942
q9	8654	6723	6661	6661
q10	8732	3812	3816	3812
q11	445	251	253	251
q12	411	224	222	222
q13	17757	2973	2926	2926
q14	270	239	244	239
q15	527	485	476	476
q16	519	381	379	379
q17	952	914	899	899
q18	8055	7424	7222	7222
q19	1391	1213	1212	1212
q20	554	331	343	331
q21	5313	4703	4757	4703
q22	350	288	285	285
Total cold run time: 113579 ms
Total hot run time: 41632 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4062	4040	4015	4015
q2	337	225	223	223
q3	2992	3008	3130	3008
q4	2033	2000	2028	2000
q5	5519	5451	5386	5386
q6	220	135	138	135
q7	2132	1814	1818	1814
q8	3314	3362	3370	3362
q9	8711	8700	8575	8575
q10	3859	4021	4053	4021
q11	550	450	477	450
q12	761	599	567	567
q13	10722	3098	3113	3098
q14	314	274	266	266
q15	524	500	493	493
q16	467	426	407	407
q17	1769	1733	1727	1727
q18	8254	7799	7733	7733
q19	1721	1734	1716	1716
q20	2052	1858	1826	1826
q21	5660	5334	5410	5334
q22	531	490	470	470
Total cold run time: 66504 ms
Total hot run time: 56626 ms

@doris-robot
Copy link

doris-robot commented Jul 30, 2024

TPC-DS: Total hot run time: 170099 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 19a35422b4a6e27469cc956a3359b9c24f950217, data reload: false

query1	913	378	363	363
query2	6305	1778	1720	1720
query3	6658	210	224	210
query4	20075	17420	17233	17233
query5	3707	497	499	497
query6	269	180	172	172
query7	4597	298	294	294
query8	256	203	197	197
query9	8525	2391	2366	2366
query10	460	303	270	270
query11	10394	10151	9989	9989
query12	123	88	91	88
query13	1661	372	390	372
query14	9127	8511	7708	7708
query15	200	162	168	162
query16	6952	412	456	412
query17	966	569	558	558
query18	1863	293	285	285
query19	235	142	141	141
query20	88	86	85	85
query21	198	101	101	101
query22	4385	4192	4039	4039
query23	33867	33684	33414	33414
query24	10342	3153	3072	3072
query25	691	408	442	408
query26	1688	160	158	158
query27	2905	276	282	276
query28	7352	2012	2002	2002
query29	1267	426	436	426
query30	238	152	154	152
query31	950	782	779	779
query32	97	56	56	56
query33	690	317	346	317
query34	906	520	516	516
query35	891	770	749	749
query36	1056	881	893	881
query37	291	85	88	85
query38	2936	2791	2764	2764
query39	865	787	810	787
query40	250	111	110	110
query41	47	44	44	44
query42	124	101	99	99
query43	476	450	434	434
query44	1168	722	726	722
query45	211	181	179	179
query46	1074	798	774	774
query47	1819	1763	1732	1732
query48	355	290	293	290
query49	935	424	415	415
query50	914	434	442	434
query51	6768	6741	6606	6606
query52	102	92	93	92
query53	248	176	177	176
query54	648	455	469	455
query55	82	73	75	73
query56	264	249	266	249
query57	1139	1028	1024	1024
query58	282	268	262	262
query59	2654	2448	2485	2448
query60	288	285	283	283
query61	97	94	97	94
query62	885	641	652	641
query63	212	181	187	181
query64	5632	1913	1874	1874
query65	3186	3097	3087	3087
query66	1338	336	358	336
query67	15542	14928	14849	14849
query68	4407	550	543	543
query69	673	368	315	315
query70	1123	1084	1064	1064
query71	436	285	280	280
query72	7258	2682	2542	2542
query73	760	325	333	325
query74	5932	5558	5699	5558
query75	3436	2695	2719	2695
query76	2957	1312	1398	1312
query77	560	305	314	305
query78	9469	8872	8900	8872
query79	1109	527	525	525
query80	900	503	516	503
query81	563	226	224	224
query82	443	125	128	125
query83	251	166	168	166
query84	262	79	80	79
query85	1214	306	362	306
query86	437	312	314	312
query87	3303	3051	3080	3051
query88	2953	2401	2422	2401
query89	386	286	291	286
query90	1706	192	193	192
query91	127	104	101	101
query92	58	52	49	49
query93	1967	601	607	601
query94	793	287	300	287
query95	384	263	266	263
query96	612	282	278	278
query97	3191	3005	3012	3005
query98	219	198	190	190
query99	1623	1302	1277	1277
Total cold run time: 261241 ms
Total hot run time: 170099 ms

@924060929 924060929 added usercase Important user case type label p0_b dev/2.1.x labels Jul 30, 2024
@doris-robot
Copy link

doris-robot commented Jul 30, 2024

ClickBench: Total hot run time: 30.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 19a35422b4a6e27469cc956a3359b9c24f950217, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.07	0.06
query5	0.48	0.48	0.47
query6	1.14	0.72	0.71
query7	0.02	0.02	0.01
query8	0.06	0.05	0.04
query9	0.57	0.51	0.50
query10	0.56	0.57	0.56
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.59
query14	0.78	0.79	0.78
query15	0.90	0.87	0.86
query16	0.35	0.36	0.35
query17	1.01	1.00	1.01
query18	0.23	0.20	0.21
query19	1.80	1.78	1.74
query20	0.01	0.02	0.01
query21	15.42	0.76	0.66
query22	4.65	7.32	1.40
query23	17.82	1.30	1.25
query24	2.26	0.23	0.21
query25	0.19	0.08	0.08
query26	0.32	0.21	0.21
query27	0.46	0.23	0.22
query28	13.17	1.00	0.96
query29	12.51	3.31	3.32
query30	0.25	0.06	0.05
query31	2.86	0.40	0.40
query32	3.25	0.48	0.48
query33	2.92	2.93	2.96
query34	15.45	4.26	4.33
query35	4.30	4.28	4.32
query36	0.68	0.49	0.48
query37	0.19	0.16	0.17
query38	0.16	0.16	0.16
query39	0.04	0.03	0.04
query40	0.16	0.13	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 108.11 s
Total hot run time: 30.03 s

@924060929
Copy link
Contributor Author

924060929 commented Jul 30, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 30, 2024

TPC-H: Total hot run time: 41299 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e577ea47aae483f5878d5f5b6b663a72464a5bd3, data reload: false

------ Round 1 ----------------------------------
q1	17716	4074	4056	4056
q2	2016	200	202	200
q3	10467	1309	1358	1309
q4	10178	807	899	807
q5	7604	2927	2944	2927
q6	216	133	138	133
q7	1025	609	601	601
q8	9443	1805	1913	1805
q9	8510	6575	6600	6575
q10	8710	3795	3844	3795
q11	436	254	255	254
q12	435	228	230	228
q13	17757	2950	2956	2950
q14	267	240	250	240
q15	523	497	498	497
q16	532	388	387	387
q17	959	923	873	873
q18	7916	7282	7196	7196
q19	1590	1199	1203	1199
q20	565	315	336	315
q21	5279	4674	4821	4674
q22	351	279	278	278
Total cold run time: 112495 ms
Total hot run time: 41299 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4077	4005	4010	4005
q2	335	225	221	221
q3	2972	3026	3122	3026
q4	1966	1983	1978	1978
q5	5569	5528	5488	5488
q6	219	139	130	130
q7	2102	1762	1824	1762
q8	3331	3382	3332	3332
q9	8643	8693	8697	8693
q10	3966	4021	4047	4021
q11	556	447	450	447
q12	754	583	604	583
q13	16682	3093	3088	3088
q14	306	287	287	287
q15	531	493	475	475
q16	447	411	422	411
q17	1772	1733	1716	1716
q18	8099	7785	7708	7708
q19	2370	1709	1700	1700
q20	2069	1837	1841	1837
q21	5621	5425	5962	5425
q22	521	450	464	450
Total cold run time: 72908 ms
Total hot run time: 56783 ms

@doris-robot
Copy link

doris-robot commented Jul 30, 2024

TPC-DS: Total hot run time: 169964 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e577ea47aae483f5878d5f5b6b663a72464a5bd3, data reload: false

query1	903	365	366	365
query2	6468	1761	1750	1750
query3	6657	217	228	217
query4	20223	17546	17078	17078
query5	3653	504	542	504
query6	286	168	183	168
query7	4583	298	287	287
query8	251	195	195	195
query9	8539	2376	2387	2376
query10	429	283	266	266
query11	10487	10035	9937	9937
query12	122	88	88	88
query13	1627	377	370	370
query14	9095	7787	7823	7787
query15	216	164	167	164
query16	6925	440	429	429
query17	967	582	565	565
query18	1924	299	288	288
query19	194	152	150	150
query20	95	87	90	87
query21	205	103	103	103
query22	4069	3938	4139	3938
query23	33357	33905	33665	33665
query24	9283	3155	3065	3065
query25	661	421	408	408
query26	1436	149	150	149
query27	3275	283	274	274
query28	7650	2002	1989	1989
query29	1046	459	439	439
query30	238	155	155	155
query31	972	759	744	744
query32	102	54	56	54
query33	686	345	334	334
query34	926	490	510	490
query35	875	737	747	737
query36	1032	905	890	890
query37	211	80	78	78
query38	2870	2802	2757	2757
query39	850	798	823	798
query40	250	112	110	110
query41	45	44	43	43
query42	123	97	104	97
query43	472	436	428	428
query44	1191	722	740	722
query45	212	184	179	179
query46	1123	797	777	777
query47	1806	1718	1702	1702
query48	368	293	289	289
query49	860	420	409	409
query50	912	428	430	428
query51	6789	6664	6734	6664
query52	105	90	90	90
query53	253	178	177	177
query54	624	467	446	446
query55	75	71	77	71
query56	277	256	262	256
query57	1136	1040	1048	1040
query58	278	255	287	255
query59	2559	2393	2534	2393
query60	302	276	264	264
query61	101	101	97	97
query62	880	672	649	649
query63	216	870	180	180
query64	5354	1900	1858	1858
query65	3140	3087	3091	3087
query66	1053	337	334	334
query67	15293	14807	14807	14807
query68	4332	549	567	549
query69	560	350	307	307
query70	1094	1086	1057	1057
query71	382	272	272	272
query72	7268	2684	2527	2527
query73	749	323	322	322
query74	6042	5576	5616	5576
query75	3407	2729	2710	2710
query76	2204	1353	1344	1344
query77	454	309	317	309
query78	9414	8912	8795	8795
query79	2780	541	525	525
query80	1015	548	514	514
query81	560	230	226	226
query82	794	134	130	130
query83	250	169	171	169
query84	269	77	84	77
query85	1542	327	303	303
query86	466	367	294	294
query87	3235	3077	3138	3077
query88	3898	2373	2413	2373
query89	386	288	282	282
query90	1765	191	189	189
query91	128	103	99	99
query92	59	53	48	48
query93	2241	590	601	590
query94	793	301	299	299
query95	386	260	258	258
query96	620	276	275	275
query97	3151	2998	3019	2998
query98	223	209	215	209
query99	1658	1271	1288	1271
Total cold run time: 261310 ms
Total hot run time: 169964 ms

@doris-robot
Copy link

doris-robot commented Jul 30, 2024

ClickBench: Total hot run time: 30.28 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e577ea47aae483f5878d5f5b6b663a72464a5bd3, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.22	0.05	0.05
query4	1.67	0.06	0.06
query5	0.48	0.48	0.48
query6	1.17	0.71	0.72
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.57	0.50	0.51
query10	0.54	0.58	0.56
query11	0.16	0.12	0.12
query12	0.15	0.13	0.13
query13	0.62	0.60	0.59
query14	0.77	0.80	0.80
query15	0.89	0.85	0.86
query16	0.35	0.36	0.35
query17	0.99	1.03	0.99
query18	0.22	0.20	0.21
query19	1.83	1.69	1.74
query20	0.01	0.01	0.02
query21	15.39	0.77	0.66
query22	4.20	7.70	1.61
query23	18.12	1.32	1.33
query24	2.26	0.22	0.22
query25	0.19	0.08	0.08
query26	0.32	0.23	0.21
query27	0.46	0.24	0.24
query28	13.15	1.00	0.98
query29	12.63	3.32	3.33
query30	0.26	0.08	0.06
query31	2.85	0.41	0.41
query32	3.23	0.50	0.48
query33	2.91	2.90	2.98
query34	15.42	4.28	4.25
query35	4.29	4.34	4.27
query36	0.68	0.47	0.47
query37	0.19	0.17	0.16
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.13	0.12
query41	0.11	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.93 s
Total hot run time: 30.28 s

@924060929
Copy link
Contributor Author

924060929 commented Jul 31, 2024

run buildall

@924060929
Copy link
Contributor Author

924060929 commented Jul 31, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 31, 2024

TPC-H: Total hot run time: 41495 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1a1b86b5ca9046ccdf73ff0045b054fea19c3231, data reload: false

------ Round 1 ----------------------------------
q1	17622	4091	4055	4055
q2	2017	218	200	200
q3	10501	1352	1355	1352
q4	10180	780	957	780
q5	7641	2947	2946	2946
q6	219	138	140	138
q7	1032	611	607	607
q8	9433	1934	1928	1928
q9	8472	6603	6626	6603
q10	8743	3807	3814	3807
q11	426	244	252	244
q12	434	235	232	232
q13	17756	2932	2947	2932
q14	266	242	241	241
q15	518	490	489	489
q16	527	395	389	389
q17	954	910	899	899
q18	8042	7263	7165	7165
q19	1546	1217	1202	1202
q20	551	327	337	327
q21	5246	4812	4680	4680
q22	352	284	279	279
Total cold run time: 112478 ms
Total hot run time: 41495 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4130	4038	4032	4032
q2	329	216	224	216
q3	2979	2990	3121	2990
q4	1984	2011	1956	1956
q5	5557	5479	5406	5406
q6	216	136	133	133
q7	2123	1782	1822	1782
q8	3338	3371	3367	3367
q9	8619	8666	8695	8666
q10	3972	4019	3955	3955
q11	543	447	465	447
q12	752	617	597	597
q13	16356	3135	3127	3127
q14	301	263	264	263
q15	546	507	480	480
q16	447	409	427	409
q17	1731	1708	1725	1708
q18	8193	7858	7576	7576
q19	1773	1730	1697	1697
q20	2041	1836	1922	1836
q21	5635	5470	5433	5433
q22	498	445	461	445
Total cold run time: 72063 ms
Total hot run time: 56521 ms

@doris-robot
Copy link

doris-robot commented Jul 31, 2024

TPC-DS: Total hot run time: 170709 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1a1b86b5ca9046ccdf73ff0045b054fea19c3231, data reload: false

query1	905	367	373	367
query2	6465	1690	1680	1680
query3	6669	210	222	210
query4	19158	17635	17338	17338
query5	3663	514	508	508
query6	271	168	165	165
query7	4610	302	285	285
query8	245	192	198	192
query9	8519	2369	2361	2361
query10	443	267	269	267
query11	10531	10038	10041	10038
query12	127	97	90	90
query13	1631	371	369	369
query14	8714	7957	7886	7886
query15	202	165	161	161
query16	6807	443	454	443
query17	949	573	562	562
query18	1929	297	290	290
query19	194	150	146	146
query20	91	92	87	87
query21	208	103	99	99
query22	4513	4425	4392	4392
query23	33772	33727	33288	33288
query24	9380	3162	3115	3115
query25	657	434	414	414
query26	1329	150	151	150
query27	2866	277	277	277
query28	7461	2012	2006	2006
query29	1011	440	436	436
query30	226	153	155	153
query31	949	813	810	810
query32	99	56	52	52
query33	660	315	310	310
query34	920	498	494	494
query35	869	763	771	763
query36	1060	878	909	878
query37	172	85	80	80
query38	2969	2849	2861	2849
query39	873	832	837	832
query40	252	114	110	110
query41	45	46	43	43
query42	123	98	97	97
query43	467	414	410	410
query44	1172	742	736	736
query45	222	179	177	177
query46	1080	806	764	764
query47	1825	1704	1707	1704
query48	360	288	288	288
query49	855	430	408	408
query50	906	421	424	421
query51	6741	6672	6649	6649
query52	102	93	90	90
query53	250	176	183	176
query54	586	446	453	446
query55	75	76	75	75
query56	276	252	243	243
query57	1144	1041	1050	1041
query58	283	274	287	274
query59	2555	2370	2308	2308
query60	301	298	288	288
query61	96	95	94	94
query62	878	671	668	668
query63	213	183	180	180
query64	4835	1918	1925	1918
query65	3138	3104	3077	3077
query66	1194	329	331	329
query67	15426	14816	14735	14735
query68	4346	550	563	550
query69	471	301	296	296
query70	1130	1057	1110	1057
query71	412	281	271	271
query72	7072	2731	2483	2483
query73	756	331	320	320
query74	6017	5643	5596	5596
query75	3360	2716	2756	2716
query76	2160	1369	1568	1369
query77	409	305	307	305
query78	9478	8987	8877	8877
query79	1956	521	528	521
query80	1145	499	553	499
query81	565	230	228	228
query82	1056	136	128	128
query83	237	169	178	169
query84	264	80	78	78
query85	1151	312	298	298
query86	434	306	315	306
query87	3310	3154	3164	3154
query88	2896	2411	2369	2369
query89	392	292	288	288
query90	1696	187	186	186
query91	123	99	103	99
query92	59	50	53	50
query93	1660	601	591	591
query94	779	287	290	287
query95	380	307	257	257
query96	600	276	277	276
query97	3196	3011	3066	3011
query98	227	198	197	197
query99	1649	1285	1295	1285
Total cold run time: 256903 ms
Total hot run time: 170709 ms

@doris-robot
Copy link

doris-robot commented Jul 31, 2024

ClickBench: Total hot run time: 30.43 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1a1b86b5ca9046ccdf73ff0045b054fea19c3231, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.49	0.49	0.47
query6	1.13	0.72	0.71
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.51
query10	0.56	0.54	0.58
query11	0.15	0.12	0.12
query12	0.15	0.13	0.12
query13	0.62	0.60	0.60
query14	0.79	0.79	0.81
query15	0.89	0.86	0.87
query16	0.36	0.36	0.35
query17	0.98	0.99	0.98
query18	0.22	0.21	0.21
query19	1.85	1.73	1.79
query20	0.01	0.01	0.01
query21	15.39	0.77	0.67
query22	4.26	6.95	1.75
query23	18.13	1.28	1.27
query24	2.27	0.22	0.23
query25	0.19	0.08	0.08
query26	0.32	0.22	0.21
query27	0.46	0.24	0.23
query28	13.16	1.03	1.00
query29	12.60	3.36	3.34
query30	0.25	0.06	0.06
query31	2.85	0.42	0.42
query32	3.22	0.49	0.49
query33	2.92	2.92	2.90
query34	15.46	4.25	4.26
query35	4.29	4.23	4.29
query36	0.68	0.48	0.49
query37	0.19	0.17	0.16
query38	0.16	0.15	0.16
query39	0.04	0.04	0.04
query40	0.17	0.14	0.13
query41	0.10	0.05	0.06
query42	0.05	0.05	0.04
query43	0.05	0.04	0.03
Total cold run time: 108.04 s
Total hot run time: 30.43 s

@924060929
Copy link
Contributor Author

924060929 commented Aug 5, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 5, 2024
@doris-robot
Copy link

doris-robot commented Aug 5, 2024

TPC-H: Total hot run time: 41721 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8fff252ec2f42293b26c67adfabdc1a575b8313d, data reload: false

------ Round 1 ----------------------------------
q1	17622	4106	4130	4106
q2	2024	203	220	203
q3	10461	1228	1291	1228
q4	10163	803	946	803
q5	7652	3025	2988	2988
q6	224	138	143	138
q7	1073	631	613	613
q8	9451	1847	1975	1847
q9	9310	6654	6604	6604
q10	8850	3844	3883	3844
q11	426	254	256	254
q12	439	234	227	227
q13	17768	2936	2939	2936
q14	268	240	251	240
q15	524	474	500	474
q16	529	396	384	384
q17	965	937	936	936
q18	8019	7342	7308	7308
q19	1378	1236	1236	1236
q20	572	330	334	330
q21	5354	4747	4808	4747
q22	351	275	286	275
Total cold run time: 113423 ms
Total hot run time: 41721 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4058	4027	4051	4027
q2	323	217	219	217
q3	3017	3017	2997	2997
q4	1880	1891	1879	1879
q5	5263	5230	5251	5230
q6	220	131	131	131
q7	2039	1713	1703	1703
q8	3187	3280	3249	3249
q9	8289	8245	8290	8245
q10	3752	3849	3836	3836
q11	561	452	457	452
q12	736	552	562	552
q13	14510	2954	2943	2943
q14	292	253	263	253
q15	517	484	478	478
q16	440	402	383	383
q17	1713	1709	1712	1709
q18	7715	7268	7187	7187
q19	1662	1697	1665	1665
q20	1972	1766	1741	1741
q21	5395	5275	5221	5221
q22	512	466	452	452
Total cold run time: 68053 ms
Total hot run time: 54550 ms

@doris-robot
Copy link

doris-robot commented Aug 5, 2024

TPC-DS: Total hot run time: 168803 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8fff252ec2f42293b26c67adfabdc1a575b8313d, data reload: false

query1	916	397	378	378
query2	6306	1676	1641	1641
query3	5961	231	226	226
query4	19936	17717	17505	17505
query5	4206	507	512	507
query6	274	177	198	177
query7	4288	296	300	296
query8	261	211	198	198
query9	8133	2374	2377	2374
query10	438	270	256	256
query11	10632	9996	10047	9996
query12	147	90	88	88
query13	1555	379	383	379
query14	9724	6828	6804	6804
query15	212	168	164	164
query16	7073	445	482	445
query17	964	552	545	545
query18	1910	284	275	275
query19	199	144	139	139
query20	90	87	84	84
query21	216	98	103	98
query22	4234	4076	4053	4053
query23	33724	32906	33065	32906
query24	9963	3053	3085	3053
query25	648	382	404	382
query26	1729	148	147	147
query27	2748	272	275	272
query28	6495	1965	1971	1965
query29	1186	442	414	414
query30	283	150	152	150
query31	936	753	761	753
query32	96	53	59	53
query33	679	309	318	309
query34	887	464	484	464
query35	845	739	731	731
query36	996	883	871	871
query37	278	79	78	78
query38	2871	2795	2743	2743
query39	857	792	811	792
query40	267	115	113	113
query41	46	44	45	44
query42	124	101	108	101
query43	467	408	413	408
query44	1178	735	726	726
query45	206	177	176	176
query46	1085	817	782	782
query47	1788	1705	1729	1705
query48	361	299	290	290
query49	1144	422	429	422
query50	925	433	442	433
query51	6721	6777	6652	6652
query52	102	91	86	86
query53	257	183	182	182
query54	628	453	446	446
query55	78	76	77	76
query56	267	262	248	248
query57	1158	1044	1052	1044
query58	307	271	280	271
query59	2577	2403	2394	2394
query60	293	273	275	273
query61	97	94	96	94
query62	910	656	675	656
query63	215	181	181	181
query64	5622	1896	1904	1896
query65	3189	3110	3096	3096
query66	1377	342	330	330
query67	15377	14741	14885	14741
query68	4199	571	580	571
query69	448	297	298	297
query70	1119	1054	1039	1039
query71	404	291	279	279
query72	7154	2702	2520	2520
query73	782	333	324	324
query74	6067	5643	5631	5631
query75	3425	2735	2731	2731
query76	2404	1243	1291	1243
query77	442	311	321	311
query78	9477	8941	8906	8906
query79	1428	541	527	527
query80	973	505	506	505
query81	578	227	227	227
query82	962	131	131	131
query83	245	180	173	173
query84	277	76	80	76
query85	1618	325	302	302
query86	406	295	306	295
query87	3331	3094	3099	3094
query88	2938	2402	2421	2402
query89	384	288	288	288
query90	1819	196	188	188
query91	126	110	114	110
query92	61	53	53	53
query93	1315	628	620	620
query94	876	299	301	299
query95	391	283	277	277
query96	601	289	291	289
query97	3263	3015	3063	3015
query98	229	209	205	205
query99	1677	1298	1311	1298
Total cold run time: 259081 ms
Total hot run time: 168803 ms

@doris-robot
Copy link

doris-robot commented Aug 5, 2024

ClickBench: Total hot run time: 29.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8fff252ec2f42293b26c67adfabdc1a575b8313d, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.48	0.48	0.50
query6	1.15	0.73	0.71
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.57	0.51	0.52
query10	0.56	0.56	0.56
query11	0.16	0.11	0.12
query12	0.15	0.13	0.13
query13	0.63	0.63	0.61
query14	0.77	0.80	0.79
query15	0.90	0.87	0.86
query16	0.36	0.36	0.35
query17	1.00	1.00	1.00
query18	0.21	0.21	0.22
query19	1.87	1.79	1.74
query20	0.01	0.01	0.00
query21	15.40	0.76	0.66
query22	4.03	8.09	1.15
query23	17.84	1.33	1.37
query24	2.24	0.22	0.21
query25	0.19	0.07	0.07
query26	0.32	0.22	0.21
query27	0.46	0.23	0.23
query28	13.18	1.01	0.98
query29	12.59	3.31	3.26
query30	0.25	0.06	0.06
query31	2.87	0.42	0.41
query32	3.25	0.49	0.48
query33	2.91	3.00	2.95
query34	15.43	4.24	4.22
query35	4.28	4.29	4.30
query36	0.68	0.48	0.49
query37	0.19	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.16	0.13	0.14
query41	0.09	0.05	0.05
query42	0.07	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 107.59 s
Total hot run time: 29.88 s

@924060929
Copy link
Contributor Author

924060929 commented Aug 5, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 5, 2024

TPC-H: Total hot run time: 41413 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ac3b060f1c04c30b7a111cabc965011587ef419, data reload: false

------ Round 1 ----------------------------------
q1	17595	4162	4071	4071
q2	2017	205	201	201
q3	10488	1300	1326	1300
q4	10154	860	936	860
q5	7636	2993	2950	2950
q6	215	136	135	135
q7	1058	624	621	621
q8	9446	1813	1957	1813
q9	8377	6538	6575	6538
q10	8742	3828	3808	3808
q11	431	253	247	247
q12	407	232	227	227
q13	17758	2918	2947	2918
q14	269	241	243	241
q15	531	486	491	486
q16	508	393	387	387
q17	959	941	922	922
q18	8061	7280	7256	7256
q19	1384	1213	1217	1213
q20	572	314	333	314
q21	5311	4632	4727	4632
q22	355	284	273	273
Total cold run time: 112274 ms
Total hot run time: 41413 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4056	4015	4006	4006
q2	331	220	216	216
q3	2990	2988	2990	2988
q4	1882	1863	1830	1830
q5	5241	5205	5199	5199
q6	212	130	128	128
q7	2048	1702	1697	1697
q8	3152	3277	3192	3192
q9	8270	8239	8271	8239
q10	3740	3829	3843	3829
q11	540	445	442	442
q12	735	578	594	578
q13	15445	2940	2963	2940
q14	291	257	255	255
q15	521	474	477	474
q16	439	394	393	393
q17	1718	1691	1691	1691
q18	7685	7322	7124	7124
q19	1652	1668	1640	1640
q20	1967	1741	1743	1741
q21	5389	5148	5229	5148
q22	509	452	470	452
Total cold run time: 68813 ms
Total hot run time: 54202 ms

@doris-robot
Copy link

doris-robot commented Aug 5, 2024

TPC-DS: Total hot run time: 168551 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ac3b060f1c04c30b7a111cabc965011587ef419, data reload: false

query1	896	381	367	367
query2	6483	1640	1643	1640
query3	6695	218	231	218
query4	20009	17460	17226	17226
query5	4287	526	521	521
query6	300	197	194	194
query7	4623	328	292	292
query8	253	206	209	206
query9	8538	2392	2391	2391
query10	441	282	278	278
query11	10600	10021	10158	10021
query12	142	85	89	85
query13	1627	376	380	376
query14	9440	6773	7524	6773
query15	204	162	165	162
query16	7088	441	439	439
query17	926	553	535	535
query18	1912	274	295	274
query19	193	145	144	144
query20	91	82	87	82
query21	201	99	96	96
query22	4240	4170	3854	3854
query23	33625	33026	32949	32949
query24	10321	3074	3021	3021
query25	679	403	403	403
query26	1749	146	151	146
query27	2900	269	274	269
query28	6933	1963	1947	1947
query29	1325	414	411	411
query30	285	150	159	150
query31	920	745	742	742
query32	101	54	54	54
query33	704	317	318	317
query34	916	484	474	474
query35	853	725	730	725
query36	974	865	836	836
query37	285	78	79	78
query38	2859	2785	2758	2758
query39	858	821	811	811
query40	285	116	111	111
query41	50	45	44	44
query42	123	101	99	99
query43	455	414	417	414
query44	1184	730	732	730
query45	211	175	174	174
query46	1085	785	767	767
query47	1789	1717	1724	1717
query48	365	288	290	288
query49	1224	424	427	424
query50	904	441	438	438
query51	6826	6697	6659	6659
query52	106	93	89	89
query53	253	181	178	178
query54	632	478	463	463
query55	75	77	77	77
query56	292	257	260	257
query57	1178	1054	1030	1030
query58	290	266	269	266
query59	2522	2397	2288	2288
query60	298	269	278	269
query61	96	96	95	95
query62	929	652	667	652
query63	220	191	187	187
query64	5885	1908	1891	1891
query65	3195	3117	3121	3117
query66	1442	329	342	329
query67	15461	14846	14762	14762
query68	4354	573	573	573
query69	438	307	304	304
query70	1166	1052	1054	1052
query71	377	281	269	269
query72	7156	2828	2643	2643
query73	766	336	330	330
query74	6007	5619	5682	5619
query75	3386	2750	2772	2750
query76	2316	1195	1301	1195
query77	456	327	328	327
query78	9386	8921	8906	8906
query79	2615	534	537	534
query80	1118	527	533	527
query81	574	234	232	232
query82	736	141	135	135
query83	257	181	183	181
query84	274	84	90	84
query85	1775	376	362	362
query86	503	316	296	296
query87	3285	3086	3085	3085
query88	3903	2436	2420	2420
query89	393	291	295	291
query90	1842	200	199	199
query91	139	115	117	115
query92	69	53	54	53
query93	2354	615	624	615
query94	900	319	306	306
query95	392	281	279	279
query96	605	280	278	278
query97	3190	3034	3027	3027
query98	213	212	207	207
query99	1636	1303	1277	1277
Total cold run time: 265432 ms
Total hot run time: 168551 ms

@doris-robot
Copy link

doris-robot commented Aug 5, 2024

ClickBench: Total hot run time: 30.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3ac3b060f1c04c30b7a111cabc965011587ef419, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.09	0.09
query5	0.49	0.48	0.49
query6	1.12	0.71	0.71
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.56	0.51	0.52
query10	0.58	0.57	0.56
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.61	0.60
query14	0.78	0.79	0.78
query15	0.93	0.85	0.86
query16	0.36	0.36	0.36
query17	0.98	0.98	1.01
query18	0.22	0.21	0.21
query19	1.83	1.79	1.74
query20	0.01	0.00	0.01
query21	15.45	0.76	0.65
query22	3.64	7.43	1.50
query23	18.07	1.30	1.34
query24	2.24	0.22	0.21
query25	0.19	0.09	0.08
query26	0.32	0.22	0.22
query27	0.46	0.23	0.23
query28	13.16	1.00	0.96
query29	12.56	3.31	3.24
query30	0.25	0.06	0.06
query31	2.86	0.42	0.41
query32	3.23	0.50	0.48
query33	2.91	2.99	2.93
query34	15.45	4.27	4.29
query35	4.30	4.28	4.32
query36	0.67	0.48	0.48
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.14	0.13
query41	0.10	0.05	0.05
query42	0.05	0.06	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.34 s
Total hot run time: 30.2 s

@924060929
Copy link
Contributor Author

924060929 commented Aug 5, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 5, 2024

TPC-H: Total hot run time: 41640 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 74f296b91de361281e16d36a300ed19a7f9ffb7a, data reload: false

------ Round 1 ----------------------------------
q1	17652	4140	4068	4068
q2	2023	204	196	196
q3	10463	1288	1295	1288
q4	10171	844	899	844
q5	9511	3016	2948	2948
q6	219	140	137	137
q7	1051	618	613	613
q8	9428	1927	1958	1927
q9	8431	6578	6628	6578
q10	8689	3848	3836	3836
q11	426	251	255	251
q12	410	224	231	224
q13	17755	2962	2959	2959
q14	271	244	241	241
q15	521	485	488	485
q16	485	408	390	390
q17	966	920	911	911
q18	7973	7331	7207	7207
q19	1377	1216	1224	1216
q20	573	324	339	324
q21	5267	4720	4726	4720
q22	353	287	277	277
Total cold run time: 114015 ms
Total hot run time: 41640 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4059	4040	4027	4027
q2	325	223	218	218
q3	3017	2994	2985	2985
q4	1864	1884	1849	1849
q5	5237	5230	5217	5217
q6	214	131	127	127
q7	2009	1677	1698	1677
q8	3160	3282	3217	3217
q9	8287	8296	8244	8244
q10	3760	3814	3843	3814
q11	548	435	447	435
q12	732	544	554	544
q13	14890	2925	2947	2925
q14	285	251	251	251
q15	515	478	469	469
q16	441	416	388	388
q17	1741	1660	1707	1660
q18	7756	7392	7254	7254
q19	1658	1660	1651	1651
q20	1955	1792	1747	1747
q21	5378	5178	5199	5178
q22	512	455	457	455
Total cold run time: 68343 ms
Total hot run time: 54332 ms

@doris-robot
Copy link

doris-robot commented Aug 5, 2024

TPC-DS: Total hot run time: 168507 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 74f296b91de361281e16d36a300ed19a7f9ffb7a, data reload: false

query1	920	387	380	380
query2	6482	1670	1656	1656
query3	6661	218	228	218
query4	19687	17267	17450	17267
query5	4301	515	516	515
query6	272	165	168	165
query7	4607	307	289	289
query8	259	199	203	199
query9	8531	2376	2362	2362
query10	436	279	254	254
query11	10502	10073	9997	9997
query12	142	93	87	87
query13	1636	392	381	381
query14	9198	7595	6891	6891
query15	202	159	166	159
query16	7119	437	443	437
query17	949	565	575	565
query18	1920	284	284	284
query19	196	150	152	150
query20	93	88	87	87
query21	206	99	99	99
query22	4126	3848	3847	3847
query23	33661	32976	32980	32976
query24	10381	3063	3042	3042
query25	695	402	396	396
query26	1732	156	151	151
query27	2897	278	279	278
query28	6889	1983	1969	1969
query29	1327	449	426	426
query30	298	155	152	152
query31	952	762	750	750
query32	110	58	58	58
query33	711	319	333	319
query34	908	486	493	486
query35	866	745	718	718
query36	1004	862	861	861
query37	303	81	78	78
query38	2880	2751	2769	2751
query39	861	823	839	823
query40	285	118	114	114
query41	53	47	46	46
query42	122	103	109	103
query43	453	424	415	415
query44	1169	738	740	738
query45	209	182	182	182
query46	1084	814	878	814
query47	1799	1707	1720	1707
query48	370	292	298	292
query49	1196	434	440	434
query50	896	449	464	449
query51	6783	6839	6641	6641
query52	106	93	92	92
query53	256	186	182	182
query54	653	466	468	466
query55	81	80	79	79
query56	376	253	263	253
query57	1114	1032	1036	1032
query58	276	272	275	272
query59	2666	2390	2325	2325
query60	299	275	276	275
query61	97	95	108	95
query62	938	649	667	649
query63	219	181	194	181
query64	5878	1949	1900	1900
query65	3134	3080	3088	3080
query66	1442	335	353	335
query67	15378	14809	14693	14693
query68	7372	573	590	573
query69	747	388	311	311
query70	1110	1026	1079	1026
query71	524	274	283	274
query72	8023	2702	2465	2465
query73	965	333	329	329
query74	6031	5752	5660	5660
query75	4502	2717	2701	2701
query76	5556	1285	1300	1285
query77	879	307	313	307
query78	9536	8864	8888	8864
query79	3322	524	529	524
query80	961	507	508	507
query81	552	228	228	228
query82	771	130	131	130
query83	319	175	171	171
query84	273	83	87	83
query85	1453	312	339	312
query86	448	297	279	279
query87	3326	3108	3092	3092
query88	3714	2408	2400	2400
query89	449	285	291	285
query90	1990	195	194	194
query91	126	101	99	99
query92	66	50	49	49
query93	4090	626	620	620
query94	925	293	303	293
query95	390	273	277	273
query96	615	281	282	281
query97	3232	3042	3082	3042
query98	223	198	194	194
query99	1621	1314	1289	1289
Total cold run time: 276363 ms
Total hot run time: 168507 ms

@doris-robot
Copy link

doris-robot commented Aug 5, 2024

ClickBench: Total hot run time: 30.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 74f296b91de361281e16d36a300ed19a7f9ffb7a, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.04
query4	1.67	0.06	0.06
query5	0.48	0.48	0.48
query6	1.12	0.71	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.58	0.50	0.51
query10	0.58	0.56	0.55
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.60	0.61	0.60
query14	0.77	0.80	0.80
query15	0.92	0.85	0.86
query16	0.36	0.37	0.35
query17	1.01	0.98	0.96
query18	0.22	0.21	0.22
query19	1.89	1.80	1.72
query20	0.01	0.02	0.02
query21	15.40	0.75	0.66
query22	4.09	7.33	1.47
query23	18.09	1.32	1.30
query24	2.27	0.22	0.22
query25	0.17	0.08	0.08
query26	0.31	0.21	0.22
query27	0.46	0.23	0.23
query28	13.15	0.99	0.97
query29	12.58	3.34	3.28
query30	0.25	0.08	0.06
query31	2.86	0.40	0.41
query32	3.24	0.50	0.48
query33	2.99	2.98	2.97
query34	15.43	4.27	4.25
query35	4.28	4.28	4.30
query36	0.67	0.48	0.47
query37	0.19	0.15	0.17
query38	0.17	0.16	0.15
query39	0.05	0.03	0.03
query40	0.17	0.13	0.13
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 107.95 s
Total hot run time: 30.14 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 6, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 6, 2024

PR approved by anyone and no changes requested.

@924060929 924060929 merged commit 6b7f21c into apache:master Aug 6, 2024
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Aug 6, 2024
@924060929
[enhancement](nereids) Support eliminate outer join by match expressi…
952adca 
…on (apache#38537)

1. enable run match expression outer of filter plan, e.g join conjunct
2. support eliminate outer join by match expression, if any arguments of
match expression is null literal

(cherry picked from commit 6b7f21c)
@924060929 924060929 mentioned this pull request Aug 6, 2024
Merged
924060929 added a commit that referenced this pull request Aug 6, 2024
@924060929
[enhancement](nereids) Support eliminate outer join by match expressi…
75fe929 
…on (#38537) (#38925)

enable run match expression outer of filter plan, e.g join conjunct
support eliminate outer join by match expression, if any arguments of match expression is null literal
@morrySnow morrySnow removed the doing label Aug 7, 2024
dataroaring pushed a commit that referenced this pull request Aug 7, 2024
@924060929 @dataroaring
[enhancement](nereids) Support eliminate outer join by match expressi…
7884204 
…on (#38537)

1. enable run match expression outer of filter plan, e.g join conjunct
2. support eliminate outer join by match expression, if any arguments of
match expression is null literal
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
@csun5285 csun5285 mentioned this pull request Sep 9, 2024
Merged
qidaye pushed a commit that referenced this pull request Sep 10, 2024
@csun5285
[fix] (Nereids) restrict the execution of match on the scan (#40532)
abf6897 
revert parts of #38537 ,restrict the execution of match on the scan
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@csun5285 @dataroaring
[fix] (Nereids) restrict the execution of match on the scan (#40532)
a233cd2 
revert parts of #38537 ,restrict the execution of match on the scan
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

@Jibing-Li Jibing-Li Jibing-Li approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged p0_b reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@924060929 @doris-robot @starocean999 @Jibing-Li @wm1581066 @dataroaring @morrySnow