Skip to content

[feat](nereids) disable join reorder if any table row count is not available #40349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
englefly merged 14 commits into from
Sep 23, 2024
Merged

[feat](nereids) disable join reorder if any table row count is not available #40349

englefly merged 14 commits into from
Sep 23, 2024

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Sep 4, 2024 •
edited
Loading

Proposed changes

join reorder
if a table is not analyzed, and its row count is not fully reported, optimizer will disable join reorder.
fully report means after a table is newly created or truncated, all tablets row count are reported.

Issue Number: close #xxx

@doris-robot
Copy link

doris-robot commented Sep 4, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

englefly commented Sep 4, 2024

run buildall

@englefly englefly changed the title [feat](nereids) update how to table row count, and disable join reorder if any table row count is not available [feat](nereids) disable join reorder if any table row count is not available Sep 4, 2024
@doris-robot
Copy link

doris-robot commented Sep 4, 2024

TPC-H: Total hot run time: 38114 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b3661cd3256aee78fe50faf5ad9461dd4d1d253, data reload: false

------ Round 1 ----------------------------------
q1	17630	4503	4338	4338
q2	2030	178	183	178
q3	11879	946	1182	946
q4	10515	779	757	757
q5	7765	2854	2810	2810
q6	227	139	135	135
q7	965	618	612	612
q8	9328	2084	2058	2058
q9	7116	6535	6571	6535
q10	6985	2241	2156	2156
q11	470	241	241	241
q12	398	227	227	227
q13	17758	3099	3063	3063
q14	280	242	223	223
q15	529	487	500	487
q16	596	526	517	517
q17	983	683	707	683
q18	7393	6929	6991	6929
q19	1401	1024	982	982
q20	678	340	332	332
q21	3960	3149	2919	2919
q22	1104	986	1011	986
Total cold run time: 109990 ms
Total hot run time: 38114 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4386	4325	4272	4272
q2	385	260	270	260
q3	2850	2629	2691	2629
q4	1911	1684	1661	1661
q5	5632	5660	5744	5660
q6	243	144	137	137
q7	2236	1854	1792	1792
q8	3358	3452	3491	3452
q9	8816	8845	8799	8799
q10	3572	3410	3359	3359
q11	623	528	496	496
q12	836	657	655	655
q13	16237	3226	3309	3226
q14	334	291	284	284
q15	546	501	477	477
q16	651	581	558	558
q17	1839	1568	1561	1561
q18	8060	7774	7865	7774
q19	1757	1741	1490	1490
q20	2158	1864	1911	1864
q21	5759	5587	5401	5401
q22	1149	1037	1071	1037
Total cold run time: 73338 ms
Total hot run time: 56844 ms

@doris-robot
Copy link

doris-robot commented Sep 4, 2024

TPC-DS: Total hot run time: 197281 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b3661cd3256aee78fe50faf5ad9461dd4d1d253, data reload: false

query1	954	405	388	388
query2	12600	8737	8792	8737
query3	10679	217	223	217
query4	33617	23231	23247	23231
query5	3565	482	483	482
query6	255	171	169	169
query7	3932	306	290	290
query8	260	210	218	210
query9	8379	2521	2497	2497
query10	444	279	275	275
query11	17524	15074	14952	14952
query12	134	101	103	101
query13	1527	401	376	376
query14	9328	7111	7521	7111
query15	264	188	185	185
query16	7870	514	481	481
query17	1604	604	587	587
query18	2025	302	294	294
query19	295	153	146	146
query20	131	116	115	115
query21	208	108	105	105
query22	4452	4363	4439	4363
query23	34267	34258	35114	34258
query24	8436	2933	2818	2818
query25	529	374	390	374
query26	677	151	152	151
query27	1972	269	271	269
query28	5907	2061	2038	2038
query29	637	406	422	406
query30	260	151	148	148
query31	984	763	760	760
query32	89	49	54	49
query33	612	302	285	285
query34	857	480	480	480
query35	849	723	722	722
query36	1087	953	942	942
query37	143	86	94	86
query38	4025	3931	3783	3783
query39	1464	1395	1433	1395
query40	195	115	109	109
query41	48	43	49	43
query42	112	95	93	93
query43	511	467	478	467
query44	1091	736	732	732
query45	202	165	163	163
query46	1095	744	731	731
query47	1895	1792	1812	1792
query48	366	304	304	304
query49	837	417	420	417
query50	804	429	413	413
query51	7073	6879	6734	6734
query52	98	91	89	89
query53	247	177	177	177
query54	794	465	452	452
query55	75	74	75	74
query56	278	254	255	254
query57	1186	1071	1087	1071
query58	259	234	245	234
query59	3024	2777	2810	2777
query60	284	268	265	265
query61	101	100	99	99
query62	763	645	653	645
query63	211	179	181	179
query64	3428	673	689	673
query65	3224	3146	3158	3146
query66	685	332	329	329
query67	15382	15383	15204	15204
query68	4330	575	557	557
query69	415	288	293	288
query70	1222	1159	1071	1071
query71	333	279	259	259
query72	6334	3998	3967	3967
query73	742	328	340	328
query74	9106	8812	8919	8812
query75	3358	2696	2681	2681
query76	2022	1047	935	935
query77	422	313	322	313
query78	9722	10912	10878	10878
query79	994	556	533	533
query80	722	546	511	511
query81	556	233	228	228
query82	286	148	154	148
query83	237	149	145	145
query84	224	74	74	74
query85	683	286	280	280
query86	325	301	297	297
query87	4385	4244	4336	4244
query88	2967	2353	2338	2338
query89	383	290	288	288
query90	1749	193	189	189
query91	122	97	98	97
query92	55	49	48	48
query93	1032	532	537	532
query94	686	291	302	291
query95	350	249	301	249
query96	584	275	269	269
query97	3271	3039	3147	3039
query98	212	199	189	189
query99	1529	1265	1261	1261
Total cold run time: 286468 ms
Total hot run time: 197281 ms

@doris-robot
Copy link

doris-robot commented Sep 4, 2024

ClickBench: Total hot run time: 31.6 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4b3661cd3256aee78fe50faf5ad9461dd4d1d253, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.03	0.04
query3	0.22	0.05	0.06
query4	1.68	0.09	0.09
query5	0.52	0.49	0.51
query6	1.13	0.75	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.54	0.48	0.49
query10	0.54	0.55	0.56
query11	0.16	0.12	0.12
query12	0.16	0.13	0.13
query13	0.62	0.59	0.60
query14	2.11	2.04	2.04
query15	0.84	0.85	0.82
query16	0.37	0.38	0.37
query17	1.06	1.02	1.06
query18	0.22	0.21	0.21
query19	1.87	1.79	1.89
query20	0.01	0.01	0.01
query21	15.40	0.67	0.65
query22	3.85	7.98	1.45
query23	18.28	1.42	1.21
query24	2.12	0.21	0.20
query25	0.15	0.07	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.27	1.02	1.00
query29	12.62	3.35	3.33
query30	0.24	0.06	0.06
query31	2.87	0.39	0.39
query32	3.25	0.49	0.47
query33	3.04	2.97	3.00
query34	17.20	4.45	4.39
query35	4.43	4.51	4.45
query36	0.66	0.50	0.48
query37	0.18	0.15	0.16
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.13	0.12
query41	0.09	0.05	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.05
Total cold run time: 110.71 s
Total hot run time: 31.6 s

@englefly
Copy link
Contributor Author

englefly commented Sep 5, 2024

run buildall

@englefly englefly marked this pull request as ready for review September 5, 2024 14:50
@doris-robot
Copy link

doris-robot commented Sep 5, 2024

TPC-H: Total hot run time: 38406 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b3661cd3256aee78fe50faf5ad9461dd4d1d253, data reload: false

------ Round 1 ----------------------------------
q1	18159	4531	4346	4346
q2	2040	190	190	190
q3	11707	960	1167	960
q4	10296	761	762	761
q5	7749	2891	2809	2809
q6	233	142	145	142
q7	964	614	593	593
q8	9333	2114	2104	2104
q9	7070	6566	6610	6566
q10	7056	2200	2261	2200
q11	480	241	239	239
q12	392	227	228	227
q13	17754	3079	3091	3079
q14	292	244	245	244
q15	542	495	475	475
q16	561	450	429	429
q17	1005	701	786	701
q18	7366	6888	6962	6888
q19	1385	1046	1095	1046
q20	698	342	328	328
q21	4003	3082	3135	3082
q22	1137	1008	997	997
Total cold run time: 110222 ms
Total hot run time: 38406 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4436	4345	4310	4310
q2	389	288	262	262
q3	2944	2670	2632	2632
q4	1940	1710	1707	1707
q5	5628	5683	5732	5683
q6	238	138	144	138
q7	2257	1863	1876	1863
q8	3350	3445	3486	3445
q9	8853	8827	8854	8827
q10	3627	3394	3410	3394
q11	598	513	505	505
q12	820	645	649	645
q13	13945	3330	3180	3180
q14	326	297	278	278
q15	541	474	506	474
q16	543	503	488	488
q17	1861	1555	1524	1524
q18	8222	7819	7937	7819
q19	1761	1640	1533	1533
q20	2155	1966	1909	1909
q21	5780	5717	5549	5549
q22	1194	1097	1041	1041
Total cold run time: 71408 ms
Total hot run time: 57206 ms

@doris-robot
Copy link

doris-robot commented Sep 5, 2024

TPC-DS: Total hot run time: 195568 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b3661cd3256aee78fe50faf5ad9461dd4d1d253, data reload: false

query1	949	403	421	403
query2	12123	8486	8618	8486
query3	10615	220	220	220
query4	32852	23314	23238	23238
query5	3569	497	504	497
query6	246	167	166	166
query7	3942	294	303	294
query8	281	218	214	214
query9	8533	2492	2499	2492
query10	420	282	281	281
query11	17586	15150	15159	15150
query12	129	102	99	99
query13	1522	375	376	375
query14	8512	7342	7198	7198
query15	223	184	172	172
query16	7897	497	489	489
query17	1562	606	568	568
query18	2164	301	289	289
query19	202	145	141	141
query20	118	106	108	106
query21	204	106	108	106
query22	4493	4133	4083	4083
query23	33936	33859	33672	33672
query24	11245	2930	2926	2926
query25	650	436	440	436
query26	1246	189	160	160
query27	2696	296	297	296
query28	7342	2126	2113	2113
query29	850	433	426	426
query30	280	161	160	160
query31	962	767	775	767
query32	98	53	56	53
query33	748	296	304	296
query34	928	498	509	498
query35	885	742	721	721
query36	1100	1004	990	990
query37	158	89	88	88
query38	4156	3824	3851	3824
query39	1420	1380	1373	1373
query40	201	112	112	112
query41	46	45	43	43
query42	110	94	95	94
query43	489	465	471	465
query44	1218	755	736	736
query45	193	161	164	161
query46	1091	766	795	766
query47	1866	1809	1782	1782
query48	384	298	287	287
query49	858	459	452	452
query50	835	436	417	417
query51	6938	6835	6826	6826
query52	99	87	86	86
query53	247	185	185	185
query54	941	460	471	460
query55	78	77	79	77
query56	277	257	265	257
query57	1211	1125	1093	1093
query58	265	231	243	231
query59	2990	2801	2734	2734
query60	300	266	266	266
query61	101	96	98	96
query62	816	656	662	656
query63	220	190	185	185
query64	5165	662	644	644
query65	3194	3169	3157	3157
query66	1301	364	341	341
query67	15583	15177	15343	15177
query68	3464	626	595	595
query69	399	279	285	279
query70	1166	1105	1133	1105
query71	344	276	278	276
query72	6271	3979	4045	3979
query73	751	324	338	324
query74	9209	8971	8932	8932
query75	3377	2712	2703	2703
query76	1946	998	1017	998
query77	489	328	319	319
query78	9715	9157	9305	9157
query79	1046	541	533	533
query80	706	522	517	517
query81	527	233	227	227
query82	234	152	147	147
query83	169	153	146	146
query84	229	78	74	74
query85	693	291	343	291
query86	307	299	290	290
query87	4396	4161	4362	4161
query88	2902	2347	2349	2347
query89	374	282	287	282
query90	1868	193	193	193
query91	128	101	102	101
query92	60	49	51	49
query93	1042	527	541	527
query94	713	308	309	308
query95	348	264	256	256
query96	592	275	266	266
query97	3221	3063	3105	3063
query98	229	199	202	199
query99	1532	1272	1311	1272
Total cold run time: 292306 ms
Total hot run time: 195568 ms

@doris-robot
Copy link

doris-robot commented Sep 5, 2024

ClickBench: Total hot run time: 30.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4b3661cd3256aee78fe50faf5ad9461dd4d1d253, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.51	0.50	0.51
query6	1.13	0.74	0.73
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.53	0.49	0.47
query10	0.53	0.56	0.52
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.60	0.59	0.59
query14	1.38	1.41	1.43
query15	0.84	0.83	0.82
query16	0.38	0.36	0.38
query17	1.08	1.01	0.97
query18	0.21	0.20	0.20
query19	1.88	1.78	1.71
query20	0.01	0.00	0.01
query21	15.40	0.66	0.66
query22	4.30	7.91	1.43
query23	18.28	1.32	1.23
query24	2.16	0.22	0.22
query25	0.16	0.08	0.08
query26	0.28	0.18	0.17
query27	0.08	0.07	0.08
query28	13.20	1.03	1.00
query29	12.63	3.36	3.27
query30	0.24	0.05	0.06
query31	2.87	0.42	0.41
query32	3.23	0.49	0.48
query33	2.99	2.98	2.99
query34	17.19	4.46	4.43
query35	4.52	4.54	4.45
query36	0.65	0.46	0.48
query37	0.19	0.16	0.15
query38	0.16	0.15	0.14
query39	0.06	0.03	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.44 s
Total hot run time: 30.75 s

@englefly
Copy link
Contributor Author

englefly commented Sep 9, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 9, 2024

TPC-H: Total hot run time: 38290 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c9998ac9f7f66ddd55e69fbe64ec9ccd434f6a0, data reload: false

------ Round 1 ----------------------------------
q1	18094	4570	4457	4457
q2	2492	204	182	182
q3	11424	1136	1025	1025
q4	10296	803	725	725
q5	7769	2881	2885	2881
q6	234	142	142	142
q7	977	632	607	607
q8	9697	2093	2097	2093
q9	7195	6549	6557	6549
q10	7033	2218	2190	2190
q11	457	242	245	242
q12	403	222	217	217
q13	17781	3072	3087	3072
q14	277	234	245	234
q15	551	484	479	479
q16	530	426	419	419
q17	983	707	737	707
q18	7387	6949	6870	6870
q19	1388	1080	1003	1003
q20	720	334	325	325
q21	4029	3098	2851	2851
q22	1108	1025	1020	1020
Total cold run time: 110825 ms
Total hot run time: 38290 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4401	4313	4304	4304
q2	379	271	287	271
q3	2902	2688	2679	2679
q4	1983	1766	1676	1676
q5	5475	5421	5415	5415
q6	225	131	133	131
q7	2140	1732	1724	1724
q8	3216	3377	3379	3377
q9	8500	8501	8513	8501
q10	3505	3209	3182	3182
q11	596	494	497	494
q12	811	600	602	600
q13	9580	3111	3071	3071
q14	311	274	268	268
q15	524	489	487	487
q16	509	461	458	458
q17	1817	1486	1471	1471
q18	7887	7495	7508	7495
q19	1686	1524	1579	1524
q20	2064	1818	1802	1802
q21	5433	5203	5325	5203
q22	1127	1044	1010	1010
Total cold run time: 65071 ms
Total hot run time: 55143 ms

@doris-robot
Copy link

doris-robot commented Sep 9, 2024

TPC-DS: Total hot run time: 186844 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c9998ac9f7f66ddd55e69fbe64ec9ccd434f6a0, data reload: false

query1	915	379	393	379
query2	6493	2012	1961	1961
query3	6651	208	218	208
query4	34318	23207	23052	23052
query5	4160	510	482	482
query6	262	172	185	172
query7	4593	295	289	289
query8	316	242	222	222
query9	8488	2486	2499	2486
query10	427	272	274	272
query11	16678	14972	14935	14935
query12	150	103	123	103
query13	1639	392	357	357
query14	11367	7168	6893	6893
query15	279	173	176	173
query16	7632	450	454	450
query17	1570	568	549	549
query18	1486	286	307	286
query19	296	148	141	141
query20	119	118	112	112
query21	208	110	102	102
query22	4596	4187	4069	4069
query23	34255	33234	33985	33234
query24	11051	2900	2786	2786
query25	624	375	398	375
query26	1172	151	154	151
query27	2346	273	272	272
query28	7242	2033	2030	2030
query29	839	437	428	428
query30	309	165	155	155
query31	1023	775	804	775
query32	102	59	62	59
query33	769	297	295	295
query34	960	479	483	479
query35	874	737	716	716
query36	1052	935	915	915
query37	163	92	91	91
query38	4007	3881	3819	3819
query39	1462	1388	1407	1388
query40	201	122	118	118
query41	51	50	49	49
query42	121	96	95	95
query43	513	471	477	471
query44	1265	767	745	745
query45	198	172	170	170
query46	1140	756	739	739
query47	1856	1792	1789	1789
query48	358	294	290	290
query49	1115	437	438	437
query50	810	429	425	425
query51	6939	6915	6831	6831
query52	100	89	90	89
query53	256	188	182	182
query54	950	466	465	465
query55	81	77	72	72
query56	276	295	264	264
query57	1181	1051	1064	1051
query58	252	224	231	224
query59	2945	2778	2724	2724
query60	297	267	271	267
query61	104	105	108	105
query62	826	661	673	661
query63	215	184	185	184
query64	4443	719	655	655
query65	3215	3185	3144	3144
query66	1356	339	338	338
query67	15643	15230	15248	15230
query68	3543	592	559	559
query69	399	281	280	280
query70	1183	1136	1126	1126
query71	338	270	271	270
query72	6438	4037	3998	3998
query73	739	325	322	322
query74	9090	8891	8755	8755
query75	3408	2743	2641	2641
query76	2016	1050	971	971
query77	512	312	315	312
query78	9652	9079	8991	8991
query79	1018	533	526	526
query80	702	504	562	504
query81	452	235	227	227
query82	249	138	142	138
query83	180	153	160	153
query84	237	80	77	77
query85	688	293	275	275
query86	322	276	306	276
query87	4396	4205	4257	4205
query88	3227	2268	2263	2263
query89	376	286	295	286
query90	1855	213	194	194
query91	125	99	149	99
query92	61	49	48	48
query93	1035	554	541	541
query94	813	299	299	299
query95	360	259	254	254
query96	586	259	267	259
query97	3231	3114	3075	3075
query98	214	201	198	198
query99	1429	1279	1247	1247
Total cold run time: 286474 ms
Total hot run time: 186844 ms

@doris-robot
Copy link

doris-robot commented Sep 9, 2024

ClickBench: Total hot run time: 31.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c9998ac9f7f66ddd55e69fbe64ec9ccd434f6a0, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.08	0.08
query5	0.51	0.50	0.50
query6	1.14	0.75	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.54	0.49	0.49
query10	0.53	0.55	0.54
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	1.34	1.41	1.42
query15	0.87	0.82	0.80
query16	0.36	0.38	0.37
query17	1.06	1.02	1.06
query18	0.21	0.21	0.20
query19	1.95	1.84	1.79
query20	0.01	0.01	0.01
query21	15.39	0.68	0.67
query22	4.37	6.62	2.23
query23	18.31	1.44	1.42
query24	2.05	0.26	0.22
query25	0.15	0.08	0.08
query26	0.27	0.18	0.18
query27	0.07	0.08	0.08
query28	13.23	1.02	1.00
query29	12.64	3.35	3.35
query30	0.25	0.06	0.05
query31	2.89	0.40	0.39
query32	3.26	0.48	0.47
query33	2.94	3.01	3.03
query34	17.06	4.40	4.34
query35	4.41	4.46	4.38
query36	0.66	0.48	0.49
query37	0.19	0.15	0.15
query38	0.15	0.16	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.07	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 110.2 s
Total hot run time: 31.91 s

@englefly
Copy link
Contributor Author

englefly commented Sep 10, 2024

run buildall

3 similar comments
@englefly
Copy link
Contributor Author

englefly commented Sep 11, 2024

run buildall

@englefly
Copy link
Contributor Author

englefly commented Sep 11, 2024

run buildall

@englefly
Copy link
Contributor Author

englefly commented Sep 11, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 11, 2024

TPC-H: Total hot run time: 38143 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a656d37037e55e4611f223e3bb02a171a9b99f11, data reload: false

------ Round 1 ----------------------------------
q1	17622	4737	4350	4350
q2	1983	331	311	311
q3	12038	954	1154	954
q4	10529	839	729	729
q5	7774	2897	2815	2815
q6	225	136	134	134
q7	952	591	600	591
q8	9326	2117	2037	2037
q9	6982	6547	6544	6544
q10	7032	2273	2216	2216
q11	457	243	248	243
q12	406	230	223	223
q13	17986	3089	3105	3089
q14	284	232	254	232
q15	537	502	489	489
q16	527	423	440	423
q17	977	661	645	645
q18	7397	6799	6921	6799
q19	1390	1077	1009	1009
q20	714	338	330	330
q21	3936	3067	2972	2972
q22	1083	1021	1008	1008
Total cold run time: 110157 ms
Total hot run time: 38143 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4392	4315	4328	4315
q2	384	288	284	284
q3	2918	2647	2687	2647
q4	1970	1723	1677	1677
q5	5674	5720	5767	5720
q6	225	136	142	136
q7	2226	1865	1830	1830
q8	3353	3437	3507	3437
q9	8820	8851	8857	8851
q10	3621	3433	3371	3371
q11	621	538	510	510
q12	850	648	683	648
q13	14919	3269	3312	3269
q14	315	283	293	283
q15	540	504	511	504
q16	540	519	491	491
q17	1869	1561	1553	1553
q18	8328	8000	8026	8000
q19	1780	1632	1546	1546
q20	2182	1891	1950	1891
q21	5785	5397	5606	5397
q22	1111	1064	1058	1058
Total cold run time: 72423 ms
Total hot run time: 57418 ms

@doris-robot
Copy link

doris-robot commented Sep 11, 2024

TPC-DS: Total hot run time: 193377 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a656d37037e55e4611f223e3bb02a171a9b99f11, data reload: false

query1	904	383	405	383
query2	6402	1895	1846	1846
query3	8658	191	199	191
query4	32025	23263	23236	23236
query5	3610	484	501	484
query6	262	163	162	162
query7	4160	300	285	285
query8	278	212	229	212
query9	8404	2483	2484	2483
query10	431	268	263	263
query11	16611	15020	15197	15020
query12	142	99	99	99
query13	1523	404	367	367
query14	9284	7290	6882	6882
query15	248	167	174	167
query16	7998	464	454	454
query17	1591	561	543	543
query18	2078	290	280	280
query19	189	150	142	142
query20	120	111	111	111
query21	208	103	102	102
query22	4608	4411	4480	4411
query23	35004	34006	34109	34006
query24	11106	2955	2966	2955
query25	666	396	403	396
query26	1226	164	170	164
query27	2419	301	282	282
query28	6973	2036	2034	2034
query29	803	394	393	393
query30	257	153	154	153
query31	968	757	722	722
query32	107	54	52	52
query33	723	289	289	289
query34	902	477	466	466
query35	864	732	734	732
query36	1060	924	957	924
query37	155	91	87	87
query38	3915	3880	3952	3880
query39	1469	1421	1405	1405
query40	205	117	119	117
query41	52	49	47	47
query42	124	98	96	96
query43	494	463	471	463
query44	1268	751	740	740
query45	195	168	167	167
query46	1110	728	751	728
query47	1853	1764	1811	1764
query48	387	298	289	289
query49	880	443	451	443
query50	838	411	426	411
query51	7009	6810	6878	6810
query52	105	88	88	88
query53	252	186	183	183
query54	878	471	462	462
query55	80	75	76	75
query56	292	281	268	268
query57	1206	1067	1095	1067
query58	248	236	252	236
query59	2974	2737	2782	2737
query60	309	279	283	279
query61	141	125	135	125
query62	778	647	665	647
query63	225	197	189	189
query64	4140	794	746	746
query65	3234	3152	3146	3146
query66	757	352	369	352
query67	15791	15298	15357	15298
query68	3036	868	861	861
query69	427	328	337	328
query70	1149	1169	1173	1169
query71	345	350	347	347
query72	6152	3659	3683	3659
query73	588	572	579	572
query74	8939	8953	8833	8833
query75	3119	2943	3004	2943
query76	1728	852	837	837
query77	443	397	403	397
query78	9338	9342	9548	9342
query79	918	875	861	861
query80	813	815	793	793
query81	452	251	256	251
query82	268	263	264	263
query83	190	189	190	189
query84	194	107	105	105
query85	610	425	390	390
query86	326	335	290	290
query87	4317	4394	4306	4306
query88	4385	4074	4027	4027
query89	369	363	362	362
query90	1008	308	304	304
query91	122	125	117	117
query92	74	76	73	73
query93	930	915	922	915
query94	420	346	334	334
query95	417	409	412	409
query96	469	468	471	468
query97	3089	3081	3100	3081
query98	228	229	223	223
query99	1402	1292	1302	1292
Total cold run time: 280441 ms
Total hot run time: 193377 ms

@doris-robot
Copy link

doris-robot commented Sep 11, 2024

ClickBench: Total hot run time: 32.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a656d37037e55e4611f223e3bb02a171a9b99f11, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.05	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.07
query5	0.53	0.49	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.05	0.05
query9	0.54	0.51	0.49
query10	0.55	0.57	0.54
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	1.39	1.41	1.42
query15	0.86	0.82	0.84
query16	0.38	0.37	0.37
query17	1.07	1.00	1.06
query18	0.17	0.18	0.17
query19	1.92	1.87	1.86
query20	0.02	0.01	0.01
query21	15.40	0.67	0.67
query22	4.18	6.74	2.59
query23	18.34	1.39	1.28
query24	2.11	0.24	0.21
query25	0.16	0.09	0.09
query26	0.26	0.18	0.17
query27	0.08	0.09	0.07
query28	13.15	1.04	1.00
query29	12.59	3.42	3.37
query30	0.24	0.06	0.05
query31	2.89	0.40	0.39
query32	3.23	0.49	0.47
query33	2.98	3.06	3.04
query34	17.02	4.48	4.46
query35	4.47	4.51	4.44
query36	0.66	0.47	0.47
query37	0.17	0.16	0.16
query38	0.18	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.13	0.13
query41	0.11	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.1 s
Total hot run time: 32.35 s

@englefly
Copy link
Contributor Author

englefly commented Sep 11, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 11, 2024

TPC-H: Total hot run time: 37910 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 12fd24384b9ed2c7fd8225a861122ea330cf5b12, data reload: false

------ Round 1 ----------------------------------
q1	17627	4796	4337	4337
q2	1997	307	302	302
q3	12136	972	1066	972
q4	10520	663	755	663
q5	7786	2885	2857	2857
q6	230	139	138	138
q7	963	597	604	597
q8	9343	2072	2066	2066
q9	7173	6565	6523	6523
q10	7048	2233	2188	2188
q11	455	242	245	242
q12	406	228	227	227
q13	19014	3077	3139	3077
q14	276	237	254	237
q15	555	492	492	492
q16	540	475	426	426
q17	1005	724	712	712
q18	7403	6958	6921	6921
q19	1444	794	783	783
q20	776	711	704	704
q21	3829	2390	2481	2390
q22	1093	1056	1057	1056
Total cold run time: 111619 ms
Total hot run time: 37910 ms

----- Round 2, with runtime_filter_mode=off -----
q1	3326	3340	3318	3318
q2	316	297	294	294
q3	2215	2219	2180	2180
q4	1348	1336	1354	1336
q5	4383	4551	4614	4551
q6	292	291	300	291
q7	1942	1894	1825	1825
q8	2423	2376	2384	2376
q9	7321	7329	7356	7329
q10	2912	2966	2865	2865
q11	542	545	530	530
q12	798	731	777	731
q13	15282	3646	3580	3580
q14	301	300	300	300
q15	551	533	519	519
q16	497	492	521	492
q17	1258	1194	1205	1194
q18	7064	7152	7101	7101
q19	967	954	977	954
q20	2025	1978	1961	1961
q21	5161	5260	5176	5176
q22	1141	1142	1140	1140
Total cold run time: 62065 ms
Total hot run time: 50043 ms

@doris-robot
Copy link

doris-robot commented Sep 11, 2024

TPC-DS: Total hot run time: 193048 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 12fd24384b9ed2c7fd8225a861122ea330cf5b12, data reload: false

query1	898	399	394	394
query2	6168	1886	1804	1804
query3	8671	194	200	194
query4	33362	23330	23207	23207
query5	5300	514	508	508
query6	277	162	155	155
query7	4163	291	286	286
query8	288	233	232	232
query9	8485	2485	2486	2485
query10	490	269	277	269
query11	17843	15056	15356	15056
query12	154	123	98	98
query13	1572	389	368	368
query14	10561	6788	7195	6788
query15	256	162	173	162
query16	7856	458	432	432
query17	1579	568	557	557
query18	2039	348	287	287
query19	330	158	153	153
query20	117	112	122	112
query21	210	109	102	102
query22	4797	4310	4321	4310
query23	34521	35710	34212	34212
query24	10909	2927	2956	2927
query25	559	404	405	404
query26	716	156	156	156
query27	2169	299	284	284
query28	6251	2109	2112	2109
query29	698	441	417	417
query30	199	153	160	153
query31	1012	823	796	796
query32	99	54	55	54
query33	689	293	296	293
query34	908	480	466	466
query35	856	716	731	716
query36	1131	907	918	907
query37	142	83	90	83
query38	4006	3926	3907	3907
query39	1437	1403	1403	1403
query40	200	110	115	110
query41	47	46	44	44
query42	113	95	92	92
query43	506	468	459	459
query44	1156	745	734	734
query45	192	177	171	171
query46	1131	750	751	750
query47	1887	1797	1828	1797
query48	364	291	283	283
query49	810	448	449	448
query50	835	408	407	407
query51	7043	6783	6833	6783
query52	104	86	90	86
query53	251	181	178	178
query54	891	469	454	454
query55	76	74	75	74
query56	300	266	260	260
query57	1175	1099	1056	1056
query58	245	237	229	229
query59	2956	2678	2724	2678
query60	304	272	282	272
query61	125	119	118	118
query62	794	665	680	665
query63	232	194	194	194
query64	3881	804	768	768
query65	3305	3207	3185	3185
query66	697	337	346	337
query67	15749	15435	15393	15393
query68	3025	555	567	555
query69	403	280	275	275
query70	1179	1033	1070	1033
query71	339	273	270	270
query72	6167	3982	4068	3982
query73	748	313	322	313
query74	9260	8924	9081	8924
query75	3406	2739	2640	2640
query76	1705	996	997	996
query77	442	317	315	315
query78	9956	10001	9302	9302
query79	1515	868	862	862
query80	993	823	815	815
query81	592	256	263	256
query82	681	265	264	264
query83	231	187	186	186
query84	237	108	104	104
query85	692	449	379	379
query86	426	307	325	307
query87	4485	4298	4345	4298
query88	4509	4100	4076	4076
query89	396	366	363	363
query90	1684	318	309	309
query91	121	121	121	121
query92	83	74	74	74
query93	1338	923	910	910
query94	707	346	352	346
query95	452	409	423	409
query96	470	470	480	470
query97	3156	3130	3113	3113
query98	232	237	231	231
query99	1477	1279	1303	1279
Total cold run time: 288194 ms
Total hot run time: 193048 ms

@doris-robot
Copy link

doris-robot commented Sep 11, 2024

ClickBench: Total hot run time: 32.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 12fd24384b9ed2c7fd8225a861122ea330cf5b12, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.07	0.08
query5	0.50	0.50	0.50
query6	1.12	0.72	0.73
query7	0.02	0.02	0.01
query8	0.06	0.05	0.05
query9	0.54	0.49	0.48
query10	0.54	0.54	0.54
query11	0.15	0.11	0.12
query12	0.15	0.13	0.13
query13	0.61	0.60	0.58
query14	1.37	1.42	1.45
query15	0.86	0.83	0.82
query16	0.37	0.36	0.38
query17	1.07	0.98	1.02
query18	0.18	0.18	0.18
query19	1.97	1.76	1.88
query20	0.01	0.01	0.01
query21	15.41	0.68	0.67
query22	4.22	6.77	2.48
query23	18.26	1.42	1.41
query24	2.13	0.23	0.22
query25	0.16	0.08	0.07
query26	0.27	0.18	0.17
query27	0.08	0.07	0.08
query28	13.22	1.03	1.01
query29	12.62	3.34	3.32
query30	0.25	0.05	0.05
query31	2.89	0.40	0.40
query32	3.23	0.48	0.47
query33	2.94	3.07	3.05
query34	17.01	4.44	4.44
query35	4.47	4.45	4.47
query36	0.67	0.49	0.47
query37	0.18	0.16	0.15
query38	0.15	0.14	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.11 s
Total hot run time: 32.2 s

morrySnow
morrySnow reviewed Sep 11, 2024
View reviewed changes
xzj7019
xzj7019 reviewed Sep 23, 2024
View reviewed changes
...ession-test/suites/nereids_rules_p0/push_down_filter/extract_from_disjunction_in_join.groovy
sql """
set runtime_filter_mode=OFF;
set disable_join_reorder=true;
set disable_join_reorder=true;
Copy link
Contributor

@xzj7019 xzj7019 Sep 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dup

@doris-robot
Copy link

doris-robot commented Sep 23, 2024

ClickBench: Total hot run time: 32.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b09ea841c140f2667cdcbf613ade46ead076fe96, data reload: false

query1	0.04	0.05	0.04
query2	0.07	0.03	0.02
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.50	0.50	0.51
query6	1.15	0.71	0.71
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.57	0.50	0.49
query10	0.56	0.58	0.55
query11	0.14	0.11	0.11
query12	0.13	0.10	0.10
query13	0.60	0.59	0.58
query14	2.97	2.97	3.02
query15	0.88	0.82	0.81
query16	0.40	0.38	0.38
query17	0.96	1.02	1.05
query18	0.20	0.19	0.20
query19	2.01	1.80	2.00
query20	0.01	0.02	0.01
query21	15.36	0.56	0.58
query22	2.25	2.04	1.68
query23	17.37	0.88	0.78
query24	2.51	1.22	0.79
query25	0.23	0.20	0.15
query26	0.30	0.13	0.14
query27	0.03	0.04	0.04
query28	11.50	1.13	1.18
query29	13.35	3.29	3.26
query30	0.24	0.05	0.06
query31	2.87	0.38	0.37
query32	3.29	0.47	0.47
query33	3.03	3.04	3.00
query34	17.05	4.39	4.34
query35	4.35	4.38	4.41
query36	0.67	0.47	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.96 s
Total hot run time: 32.38 s

@englefly
Copy link
Contributor Author

englefly commented Sep 23, 2024

run cloud_p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 23, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 23, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 23, 2024

PR approved by anyone and no changes requested.

@englefly englefly merged commit ddc5592 into apache:master Sep 23, 2024
@englefly englefly deleted the table-row-0903E branch September 23, 2024 07:09
@englefly englefly mentioned this pull request Sep 27, 2024
Merged
englefly added a commit that referenced this pull request Sep 29, 2024
@englefly
[feat](nereids) disable join reorder if any table row count is not av…
203d25d 
…ailable #40349 (2.0) (#41425)

## Proposed changes
pick #40349
Issue Number: close #xxx

<!--Describe your changes.-->
@englefly englefly mentioned this pull request Oct 18, 2024
Merged
englefly added a commit to englefly/incubator-doris that referenced this pull request Oct 22, 2024
@englefly
[feat](nereids) disable join reorder if any table row count is not av…
97acac6 
…ailable (apache#40349)

 join reorder
if a table is not analyzed, and its row count is not fully reported,
optimizer will disable join reorder.
fully report means after a table is newly created or truncated, all
tablets row count are reported.

Issue Number: close #xxx

<!--Describe your changes.-->
@englefly englefly mentioned this pull request Oct 22, 2024
Merged
englefly added a commit that referenced this pull request Oct 22, 2024
@englefly
[Pick](nereids) pick 40529 41464 40349 (#42073)
9c4ce73 
## Proposed changes
pick  #40529 #41464 #40349

Issue Number: close #xxx

<!--Describe your changes.-->
englefly added a commit that referenced this pull request Oct 22, 2024
@englefly
[Pick](nereids)table row related Prs 3.0 (#42237)
e3efb14 
## Proposed changes
pick #41464 #40529 #40349 #39222
Issue Number: close #xxx

<!--Describe your changes.-->
@englefly englefly mentioned this pull request Jul 24, 2025
Merged
16 tasks
englefly added a commit that referenced this pull request Aug 7, 2025
@englefly
[fix](nereids) fix bug in StatsCalculator.disableJoinReorderIfStatsIn…
7ebe2b3 
…valid when rowCount is not aviable (#53834)

### What problem does this PR solve?
this bug is introduced by pr #41790.

when any table row count is not available, disable join reorder.
this is implemented in pr #40349 , but pr #41790 breaks this function.
freemandealer pushed a commit to freemandealer/doris that referenced this pull request Aug 7, 2025
@englefly @freemandealer
[fix](nereids) fix bug in StatsCalculator.disableJoinReorderIfStatsIn…
51965a9 
…valid when rowCount is not aviable (apache#53834)

### What problem does this PR solve?
this bug is introduced by pr apache#41790.

when any table row count is not available, disable join reorder.
this is implemented in pr apache#40349 , but pr apache#41790 breaks this function.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow left review comments

@starocean999 starocean999 starocean999 approved these changes

@xzj7019 xzj7019 xzj7019 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.16-merged dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@englefly @doris-robot @starocean999 @morrySnow @xzj7019 @wm1581066