Skip to content

[improvement](external)add some improvements for external scan #38946

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 3 commits into from
Nov 3, 2024
Merged

[improvement](external)add some improvements for external scan #38946

morningman merged 3 commits into from
Nov 3, 2024

Conversation

@wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Aug 6, 2024

Proposed changes

  1. add session variable: use_consistent_hash_for_external_scan, which can specify consistent hash for external scan.
  2. add session variable: ignore_split_type, which can ignore splits of the specified type, use for performance tuning.
  3. add split weight for paimon split with consistent hash.
  4. add executeFilter for paimon jni split.

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Aug 6, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 6, 2024
@doris-robot
Copy link

doris-robot commented Aug 6, 2024

TPC-H: Total hot run time: 41239 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b4fb126d89f1559210c03bc875ed499add09209, data reload: false

------ Round 1 ----------------------------------
q1	17647	4076	4010	4010
q2	2022	200	198	198
q3	10452	1233	1284	1233
q4	10181	778	913	778
q5	7599	2961	2943	2943
q6	222	140	139	139
q7	1058	614	610	610
q8	9424	1872	1924	1872
q9	8386	6603	6636	6603
q10	8735	3814	3803	3803
q11	428	251	244	244
q12	429	225	222	222
q13	17765	2943	2964	2943
q14	274	245	242	242
q15	517	483	499	483
q16	524	392	385	385
q17	948	920	894	894
q18	7911	7276	7145	7145
q19	1448	1210	1203	1203
q20	559	332	350	332
q21	5284	4677	4673	4673
q22	345	284	285	284
Total cold run time: 112158 ms
Total hot run time: 41239 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4045	4123	4078	4078
q2	330	231	221	221
q3	2975	2953	3047	2953
q4	2001	1940	2025	1940
q5	5464	5495	5402	5402
q6	219	130	132	130
q7	2182	1757	1771	1757
q8	3314	3340	3339	3339
q9	8642	8604	8691	8604
q10	3944	4041	3950	3950
q11	548	445	449	445
q12	796	626	579	579
q13	16416	3109	3105	3105
q14	293	284	276	276
q15	538	485	503	485
q16	473	428	420	420
q17	1739	1719	1748	1719
q18	8096	7959	7896	7896
q19	4070	1693	1696	1693
q20	2147	1866	1830	1830
q21	5627	5537	5160	5160
q22	564	494	477	477
Total cold run time: 74423 ms
Total hot run time: 56459 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

TPC-DS: Total hot run time: 170008 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b4fb126d89f1559210c03bc875ed499add09209, data reload: false

query1	911	379	369	369
query2	6470	1710	1718	1710
query3	6668	209	224	209
query4	20154	17634	17482	17482
query5	3646	546	542	542
query6	282	197	180	180
query7	4596	301	301	301
query8	260	199	195	195
query9	8523	2393	2463	2393
query10	431	281	270	270
query11	10387	9846	10005	9846
query12	117	89	87	87
query13	1610	370	366	366
query14	9245	7012	7687	7012
query15	205	160	161	160
query16	6906	458	457	457
query17	948	559	547	547
query18	1897	274	294	274
query19	192	143	143	143
query20	92	84	85	84
query21	204	97	94	94
query22	4326	4092	3969	3969
query23	33796	33655	33513	33513
query24	9287	3066	3119	3066
query25	665	420	411	411
query26	1604	152	154	152
query27	3179	296	291	291
query28	7627	2015	2017	2015
query29	1133	488	449	449
query30	239	161	153	153
query31	960	802	824	802
query32	113	63	57	57
query33	690	324	363	324
query34	942	500	513	500
query35	840	756	762	756
query36	1049	912	868	868
query37	167	82	79	79
query38	2866	2788	2766	2766
query39	904	805	809	805
query40	249	116	114	114
query41	47	43	44	43
query42	122	100	104	100
query43	476	443	419	419
query44	1178	739	724	724
query45	218	180	180	180
query46	1086	805	790	790
query47	1811	1711	1729	1711
query48	366	300	289	289
query49	947	420	439	420
query50	894	446	451	446
query51	6749	6748	6629	6629
query52	107	89	90	89
query53	259	186	200	186
query54	611	448	462	448
query55	76	78	74	74
query56	272	249	256	249
query57	1122	1008	1049	1008
query58	272	272	252	252
query59	2524	2332	2421	2332
query60	286	275	287	275
query61	96	91	91	91
query62	885	661	656	656
query63	216	191	186	186
query64	5697	2018	1892	1892
query65	3162	3122	3100	3100
query66	1321	335	332	332
query67	15119	14676	14740	14676
query68	4325	549	580	549
query69	441	314	310	310
query70	1094	1119	1031	1031
query71	388	290	283	283
query72	7034	2733	2700	2700
query73	772	327	330	327
query74	5998	5682	5630	5630
query75	3367	2733	2783	2733
query76	2063	1221	1278	1221
query77	436	321	324	321
query78	9556	9044	8947	8947
query79	2742	546	539	539
query80	1158	541	537	537
query81	559	234	234	234
query82	915	142	132	132
query83	250	177	181	177
query84	272	85	84	84
query85	1342	385	362	362
query86	465	311	333	311
query87	3286	3097	3091	3091
query88	3975	2432	2447	2432
query89	391	303	290	290
query90	1784	200	202	200
query91	143	113	116	113
query92	66	52	52	52
query93	2175	627	621	621
query94	804	316	301	301
query95	404	412	279	279
query96	642	291	279	279
query97	3210	3029	3059	3029
query98	248	199	190	190
query99	1613	1295	1296	1295
Total cold run time: 262217 ms
Total hot run time: 170008 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

ClickBench: Total hot run time: 29.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b4fb126d89f1559210c03bc875ed499add09209, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.03
query3	0.22	0.04	0.04
query4	1.68	0.06	0.06
query5	0.51	0.48	0.48
query6	1.16	0.71	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.57	0.51	0.52
query10	0.57	0.56	0.56
query11	0.15	0.12	0.12
query12	0.15	0.13	0.12
query13	0.60	0.61	0.60
query14	0.77	0.79	0.80
query15	0.90	0.86	0.88
query16	0.36	0.35	0.37
query17	1.00	1.01	0.99
query18	0.22	0.21	0.22
query19	1.81	1.73	1.81
query20	0.01	0.01	0.03
query21	15.41	0.78	0.67
query22	3.80	6.73	1.28
query23	18.00	1.28	1.20
query24	2.26	0.22	0.22
query25	0.18	0.08	0.08
query26	0.32	0.21	0.21
query27	0.46	0.24	0.23
query28	13.20	1.00	0.97
query29	12.59	3.32	3.27
query30	0.25	0.06	0.05
query31	2.88	0.40	0.41
query32	3.25	0.48	0.48
query33	2.93	2.94	3.02
query34	15.45	4.27	4.33
query35	4.32	4.30	4.30
query36	0.68	0.50	0.48
query37	0.19	0.16	0.17
query38	0.16	0.16	0.15
query39	0.04	0.03	0.04
query40	0.16	0.15	0.14
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.61 s
Total hot run time: 29.93 s

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Aug 6, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

TPC-H: Total hot run time: 41919 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45279ef309af345e041d50e494b64b37e644000e, data reload: false

------ Round 1 ----------------------------------
q1	18203	4230	4191	4191
q2	2174	203	212	203
q3	10454	1268	1352	1268
q4	10192	887	1015	887
q5	7628	3047	2992	2992
q6	225	140	140	140
q7	1061	625	613	613
q8	9438	1972	1966	1966
q9	8549	6694	6597	6597
q10	8716	3867	3856	3856
q11	434	249	257	249
q12	417	229	226	226
q13	17771	2956	2933	2933
q14	282	252	238	238
q15	525	483	477	477
q16	492	397	385	385
q17	988	945	928	928
q18	8089	7392	7234	7234
q19	1380	1230	1247	1230
q20	583	330	331	330
q21	5292	4688	4767	4688
q22	348	290	288	288
Total cold run time: 113241 ms
Total hot run time: 41919 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4067	4015	4011	4011
q2	336	222	230	222
q3	2996	3038	3004	3004
q4	1923	1899	1873	1873
q5	5287	5259	5277	5259
q6	220	133	134	133
q7	2032	1682	1731	1682
q8	3190	3328	3269	3269
q9	8359	8331	8285	8285
q10	3742	3879	3887	3879
q11	551	462	453	453
q12	704	566	557	557
q13	16606	2974	2959	2959
q14	286	253	274	253
q15	513	482	471	471
q16	463	401	400	400
q17	1736	1712	1686	1686
q18	7852	7435	7218	7218
q19	1689	1678	1666	1666
q20	1956	1746	1729	1729
q21	5377	5238	5227	5227
q22	512	433	451	433
Total cold run time: 70397 ms
Total hot run time: 54669 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

TPC-DS: Total hot run time: 168283 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45279ef309af345e041d50e494b64b37e644000e, data reload: false

query1	926	378	377	377
query2	6487	1701	1580	1580
query3	6681	211	217	211
query4	18950	17403	17194	17194
query5	4333	502	525	502
query6	301	168	176	168
query7	4605	318	289	289
query8	251	194	197	194
query9	8492	2374	2451	2374
query10	471	282	275	275
query11	10563	9952	10145	9952
query12	137	88	89	88
query13	1647	381	385	381
query14	9994	7647	7091	7091
query15	209	171	189	171
query16	7098	448	430	430
query17	940	550	538	538
query18	1841	276	276	276
query19	190	143	141	141
query20	94	83	87	83
query21	204	103	102	102
query22	4131	3911	3911	3911
query23	33669	32895	32842	32842
query24	10274	3082	3061	3061
query25	687	370	388	370
query26	1736	149	152	149
query27	2865	284	281	281
query28	6871	1983	1962	1962
query29	1274	422	412	412
query30	294	149	149	149
query31	944	776	759	759
query32	108	56	59	56
query33	691	306	317	306
query34	920	487	486	486
query35	856	736	736	736
query36	984	852	854	852
query37	294	80	83	80
query38	2864	2799	2740	2740
query39	899	810	825	810
query40	290	117	114	114
query41	51	47	48	47
query42	125	103	108	103
query43	467	408	406	406
query44	1184	729	734	729
query45	212	180	179	179
query46	1089	835	802	802
query47	1799	1712	1705	1705
query48	362	303	302	302
query49	1217	433	437	433
query50	921	445	438	438
query51	6772	6716	6666	6666
query52	103	88	90	88
query53	276	185	180	180
query54	666	465	465	465
query55	78	77	76	76
query56	298	263	278	263
query57	1140	1078	1056	1056
query58	288	286	277	277
query59	2555	2461	2234	2234
query60	296	279	286	279
query61	123	115	117	115
query62	938	666	681	666
query63	232	185	181	181
query64	5970	1970	1962	1962
query65	3177	3114	3145	3114
query66	1434	339	346	339
query67	15105	14827	14610	14610
query68	4512	587	593	587
query69	501	297	339	297
query70	1100	1076	1071	1071
query71	448	286	284	284
query72	8082	2881	2483	2483
query73	775	344	339	339
query74	6047	5642	5604	5604
query75	3759	2750	2722	2722
query76	2644	1206	1274	1206
query77	664	318	327	318
query78	9630	8858	8819	8819
query79	2933	537	538	537
query80	1286	505	522	505
query81	556	228	226	226
query82	762	132	132	132
query83	377	173	191	173
query84	280	82	81	81
query85	2004	342	297	297
query86	469	296	304	296
query87	3278	3101	3129	3101
query88	3906	2432	2445	2432
query89	414	293	289	289
query90	1931	196	197	196
query91	127	99	98	98
query92	68	50	51	50
query93	4624	613	628	613
query94	940	296	303	296
query95	378	271	275	271
query96	628	285	281	281
query97	3234	3000	3039	3000
query98	210	196	197	196
query99	1592	1287	1275	1275
Total cold run time: 270102 ms
Total hot run time: 168283 ms

@doris-robot
Copy link

doris-robot commented Aug 6, 2024

ClickBench: Total hot run time: 30.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 45279ef309af345e041d50e494b64b37e644000e, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.07
query5	0.49	0.49	0.48
query6	1.15	0.71	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.57	0.51	0.52
query10	0.59	0.56	0.56
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.61	0.60	0.60
query14	0.77	0.79	0.80
query15	0.90	0.87	0.86
query16	0.36	0.35	0.35
query17	0.99	1.01	0.98
query18	0.22	0.22	0.21
query19	1.89	1.73	1.74
query20	0.01	0.01	0.01
query21	15.40	0.74	0.64
query22	3.98	7.11	1.56
query23	17.86	1.42	1.35
query24	2.23	0.23	0.22
query25	0.18	0.09	0.08
query26	0.31	0.22	0.21
query27	0.46	0.25	0.23
query28	13.20	0.99	0.97
query29	12.50	3.22	3.21
query30	0.26	0.06	0.06
query31	2.88	0.41	0.40
query32	3.22	0.50	0.48
query33	2.93	2.98	2.97
query34	15.42	4.33	4.25
query35	4.32	4.28	4.32
query36	0.68	0.48	0.49
query37	0.19	0.17	0.16
query38	0.16	0.16	0.15
query39	0.04	0.04	0.03
query40	0.16	0.13	0.12
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.51 s
Total hot run time: 30.26 s

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Aug 6, 2024

run p1

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Aug 7, 2024

run p0

@wuwenchi wuwenchi force-pushed the improve-external-table-scan branch from 45279ef to 5e9efaa Compare August 7, 2024 11:56
@wuwenchi
Copy link
Contributor Author

wuwenchi commented Aug 7, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 7, 2024

TPC-H: Total hot run time: 41954 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e9efaae42650d27014aaab3fb982bce37851ce0, data reload: false

------ Round 1 ----------------------------------
q1	18973	4229	4208	4208
q2	2482	208	210	208
q3	11711	1327	1337	1327
q4	10570	880	956	880
q5	7916	3098	2988	2988
q6	236	137	137	137
q7	1050	614	633	614
q8	9451	1910	1953	1910
q9	8550	6734	6622	6622
q10	8711	3855	3841	3841
q11	423	253	247	247
q12	410	232	235	232
q13	17763	2941	2939	2939
q14	268	244	239	239
q15	521	479	488	479
q16	532	409	410	409
q17	989	943	935	935
q18	8313	7365	7220	7220
q19	1432	1230	1221	1221
q20	575	317	324	317
q21	5333	4729	4709	4709
q22	353	272	278	272
Total cold run time: 116562 ms
Total hot run time: 41954 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4097	4010	4040	4010
q2	328	225	216	216
q3	3040	3004	3002	3002
q4	1906	1897	1840	1840
q5	5258	5292	5242	5242
q6	217	125	126	125
q7	2049	1664	1732	1664
q8	3195	3262	3259	3259
q9	8309	8296	8295	8295
q10	3759	3830	3841	3830
q11	564	455	456	455
q12	728	572	593	572
q13	14201	2967	2922	2922
q14	283	256	274	256
q15	530	480	486	480
q16	435	425	381	381
q17	1723	1692	1701	1692
q18	7696	7331	7260	7260
q19	1673	1699	1679	1679
q20	1985	1761	1738	1738
q21	5528	5132	5278	5132
q22	512	466	464	464
Total cold run time: 68016 ms
Total hot run time: 54514 ms

@doris-robot
Copy link

doris-robot commented Aug 7, 2024

TPC-DS: Total hot run time: 168002 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e9efaae42650d27014aaab3fb982bce37851ce0, data reload: false

query1	914	374	369	369
query2	6468	1686	1703	1686
query3	6672	213	225	213
query4	19714	17397	17638	17397
query5	4317	506	533	506
query6	285	173	165	165
query7	4588	293	289	289
query8	248	205	193	193
query9	8535	2393	2286	2286
query10	425	271	261	261
query11	10462	9946	10106	9946
query12	131	86	86	86
query13	1625	393	369	369
query14	9346	6778	6128	6128
query15	206	161	169	161
query16	7111	431	441	431
query17	927	553	543	543
query18	1914	276	276	276
query19	186	140	144	140
query20	93	84	87	84
query21	204	98	104	98
query22	4139	3990	3864	3864
query23	33699	33060	33042	33042
query24	10337	3093	3081	3081
query25	694	407	415	407
query26	1803	158	158	158
query27	3009	279	273	273
query28	6987	1931	1918	1918
query29	1360	442	439	439
query30	283	154	150	150
query31	920	766	750	750
query32	102	57	55	55
query33	713	308	337	308
query34	908	477	480	477
query35	860	736	719	719
query36	988	857	867	857
query37	304	88	83	83
query38	2870	2794	2770	2770
query39	879	812	813	812
query40	289	117	114	114
query41	50	49	51	49
query42	130	98	104	98
query43	484	429	423	423
query44	1186	744	737	737
query45	213	184	180	180
query46	1087	830	815	815
query47	1855	1708	1740	1708
query48	373	300	298	298
query49	1205	443	458	443
query50	905	441	430	430
query51	6870	6714	6585	6585
query52	103	90	89	89
query53	258	187	180	180
query54	625	463	465	463
query55	80	76	76	76
query56	280	270	267	267
query57	1171	1044	1043	1043
query58	278	268	279	268
query59	2567	2294	2507	2294
query60	319	286	286	286
query61	117	117	116	116
query62	930	670	647	647
query63	218	183	184	183
query64	5978	1974	1953	1953
query65	3156	3128	3108	3108
query66	1454	347	337	337
query67	15261	14826	14735	14735
query68	6032	572	681	572
query69	708	364	316	316
query70	1118	1076	1062	1062
query71	489	281	293	281
query72	8043	2697	2479	2479
query73	910	331	350	331
query74	6074	5604	5729	5604
query75	4175	2741	2736	2736
query76	3679	1258	1286	1258
query77	685	309	316	309
query78	9494	8999	8894	8894
query79	3290	529	527	527
query80	2145	501	510	501
query81	570	228	255	228
query82	1247	135	132	132
query83	273	173	174	173
query84	268	94	82	82
query85	1507	354	301	301
query86	448	267	268	267
query87	3270	3100	3210	3100
query88	3765	2493	2496	2493
query89	426	292	287	287
query90	1924	198	195	195
query91	129	119	103	103
query92	62	52	51	51
query93	4511	614	615	614
query94	903	289	302	289
query95	383	265	267	265
query96	626	289	281	281
query97	3207	3066	3077	3066
query98	220	198	195	195
query99	1616	1295	1332	1295
Total cold run time: 274865 ms
Total hot run time: 168002 ms

@doris-robot
Copy link

doris-robot commented Aug 7, 2024

ClickBench: Total hot run time: 30.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5e9efaae42650d27014aaab3fb982bce37851ce0, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.48	0.48	0.49
query6	1.14	0.71	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.54	0.50
query10	0.58	0.57	0.58
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.59
query14	0.77	0.80	0.78
query15	0.91	0.87	0.85
query16	0.35	0.34	0.36
query17	1.04	0.99	1.02
query18	0.22	0.21	0.21
query19	1.85	1.77	1.75
query20	0.03	0.02	0.01
query21	15.44	0.73	0.66
query22	3.97	7.53	1.23
query23	17.86	1.36	1.35
query24	2.26	0.22	0.21
query25	0.18	0.09	0.08
query26	0.32	0.22	0.21
query27	0.46	0.24	0.22
query28	13.19	0.98	0.97
query29	12.61	3.32	3.30
query30	0.26	0.07	0.06
query31	2.85	0.40	0.41
query32	3.25	0.48	0.48
query33	2.92	2.97	2.98
query34	15.44	4.27	4.28
query35	4.28	4.30	4.29
query36	0.67	0.50	0.48
query37	0.19	0.17	0.18
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.15	0.13	0.14
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.62 s
Total hot run time: 30.04 s

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Aug 9, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 9, 2024

TPC-H: Total hot run time: 39598 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7ba0f8ef6ab8abb8f77fa6a25e8942c7bf832add, data reload: false

------ Round 1 ----------------------------------
q1	17647	4299	4253	4253
q2	2018	174	185	174
q3	10464	1150	1112	1112
q4	10154	753	724	724
q5	7493	2505	2497	2497
q6	228	140	135	135
q7	994	595	622	595
q8	9225	1917	1882	1882
q9	8838	6580	6598	6580
q10	7045	2231	2207	2207
q11	480	242	248	242
q12	468	231	223	223
q13	17764	2990	2972	2972
q14	283	229	239	229
q15	525	474	480	474
q16	504	381	384	381
q17	968	644	710	644
q18	7928	7416	7447	7416
q19	3332	983	979	979
q20	710	319	341	319
q21	5394	4711	4581	4581
q22	1085	1029	979	979
Total cold run time: 113547 ms
Total hot run time: 39598 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4397	4310	4276	4276
q2	381	266	268	266
q3	2862	2610	2629	2610
q4	1979	1750	1780	1750
q5	5528	5699	5534	5534
q6	224	132	131	131
q7	2126	1719	1732	1719
q8	3334	3469	3413	3413
q9	8828	8796	8918	8796
q10	3548	3262	3279	3262
q11	611	483	507	483
q12	801	604	622	604
q13	16969	3168	3172	3168
q14	320	281	293	281
q15	523	508	497	497
q16	490	443	446	443
q17	1886	1554	1508	1508
q18	7962	7932	7831	7831
q19	4019	1539	1595	1539
q20	2173	1885	1893	1885
q21	5560	5337	5480	5337
q22	1101	1022	1050	1022
Total cold run time: 75622 ms
Total hot run time: 56355 ms

@doris-robot
Copy link

doris-robot commented Aug 9, 2024

TPC-DS: Total hot run time: 202328 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7ba0f8ef6ab8abb8f77fa6a25e8942c7bf832add, data reload: false

query1	911	372	361	361
query2	6425	1960	1823	1823
query3	6642	216	215	215
query4	30660	23208	23055	23055
query5	3636	486	507	486
query6	268	187	162	162
query7	4575	300	305	300
query8	228	190	189	189
query9	8556	2463	2431	2431
query10	541	482	501	482
query11	15771	15098	15063	15063
query12	140	95	94	94
query13	1620	366	358	358
query14	10248	7286	7738	7286
query15	265	223	225	223
query16	7248	490	471	471
query17	1310	547	541	541
query18	1258	282	290	282
query19	197	149	150	149
query20	114	108	108	108
query21	203	109	105	105
query22	4176	4507	4402	4402
query23	34977	33994	33822	33822
query24	11451	2612	2700	2612
query25	584	386	406	386
query26	1028	165	168	165
query27	2282	293	297	293
query28	6019	2063	2023	2023
query29	762	409	412	409
query30	265	151	152	151
query31	969	739	733	733
query32	96	57	57	57
query33	759	276	289	276
query34	886	472	489	472
query35	944	896	846	846
query36	1092	928	933	928
query37	142	77	80	77
query38	4220	4187	4186	4186
query39	1438	1386	1375	1375
query40	207	120	118	118
query41	46	41	45	41
query42	118	100	97	97
query43	495	461	485	461
query44	1223	732	729	729
query45	238	197	196	196
query46	1066	763	715	715
query47	1844	1791	1765	1765
query48	369	293	295	293
query49	841	422	429	422
query50	788	414	416	414
query51	6781	6810	6694	6694
query52	100	92	93	92
query53	255	181	178	178
query54	945	455	455	455
query55	77	73	74	73
query56	284	248	252	248
query57	1152	1071	1039	1039
query58	236	240	244	240
query59	2834	2743	2993	2743
query60	305	275	319	275
query61	97	90	92	90
query62	798	636	627	627
query63	207	185	186	185
query64	9194	2249	1812	1812
query65	3198	3147	3139	3139
query66	803	330	335	330
query67	15315	14747	14805	14747
query68	4480	549	544	544
query69	426	373	406	373
query70	1212	1113	1177	1113
query71	420	279	280	279
query72	18966	17190	16554	16554
query73	776	333	334	333
query74	9266	8802	8818	8802
query75	3338	2713	2721	2713
query76	2652	1038	999	999
query77	634	317	316	316
query78	10511	9125	8922	8922
query79	4746	534	543	534
query80	1456	507	511	507
query81	591	218	224	218
query82	760	139	141	139
query83	184	148	145	145
query84	269	87	76	76
query85	1092	276	354	276
query86	398	304	293	293
query87	4839	4609	4652	4609
query88	4884	2478	2529	2478
query89	432	288	288	288
query90	1911	196	202	196
query91	121	96	94	94
query92	62	51	50	50
query93	5026	538	541	538
query94	826	309	308	308
query95	368	273	266	266
query96	612	282	277	277
query97	3264	3032	3063	3032
query98	224	198	200	198
query99	1556	1243	1273	1243
Total cold run time: 309024 ms
Total hot run time: 202328 ms

@doris-robot
Copy link

doris-robot commented Aug 9, 2024

ClickBench: Total hot run time: 30.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7ba0f8ef6ab8abb8f77fa6a25e8942c7bf832add, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.08
query5	0.49	0.47	0.48
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.57	0.48	0.49
query10	0.54	0.53	0.54
query11	0.16	0.11	0.12
query12	0.16	0.12	0.12
query13	0.60	0.61	0.59
query14	0.77	0.79	0.79
query15	0.85	0.83	0.82
query16	0.35	0.35	0.37
query17	1.05	0.98	0.99
query18	0.24	0.22	0.24
query19	1.91	1.77	1.82
query20	0.03	0.01	0.00
query21	15.40	0.79	0.67
query22	3.96	8.09	1.86
query23	18.29	1.35	1.24
query24	2.05	0.25	0.22
query25	0.14	0.08	0.08
query26	0.30	0.22	0.21
query27	0.46	0.23	0.23
query28	13.32	1.02	1.00
query29	12.58	3.34	3.30
query30	0.23	0.05	0.04
query31	2.88	0.39	0.38
query32	3.28	0.48	0.47
query33	2.91	3.01	2.95
query34	17.09	4.43	4.43
query35	4.43	4.46	4.45
query36	0.66	0.48	0.46
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.14	0.13	0.12
query41	0.08	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.62 s
Total hot run time: 30.79 s

@wuwenchi wuwenchi force-pushed the improve-external-table-scan branch from 7ba0f8e to aec6027 Compare August 12, 2024 02:04
@wuwenchi
Copy link
Contributor Author

wuwenchi commented Aug 12, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 12, 2024

TPC-H: Total hot run time: 39937 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aec60270acb6995c5dd55134a264548d16f15e38, data reload: false

------ Round 1 ----------------------------------
q1	17628	4352	4312	4312
q2	2025	178	194	178
q3	10450	1155	1138	1138
q4	10141	745	785	745
q5	7511	2529	2494	2494
q6	222	141	140	140
q7	981	601	599	599
q8	9297	1894	1944	1894
q9	8909	6536	6609	6536
q10	7059	2122	2144	2122
q11	456	253	250	250
q12	448	229	228	228
q13	18922	3003	3012	3003
q14	272	240	243	240
q15	521	486	488	486
q16	521	396	401	396
q17	974	633	673	633
q18	8078	7555	7468	7468
q19	3432	1059	1066	1059
q20	650	318	324	318
q21	5362	4835	4680	4680
q22	1130	1018	1029	1018
Total cold run time: 114989 ms
Total hot run time: 39937 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4478	4250	4247	4247
q2	392	258	268	258
q3	2899	2610	2762	2610
q4	2057	1766	1768	1766
q5	5714	5465	5436	5436
q6	224	138	133	133
q7	2098	1913	1788	1788
q8	3307	3461	3534	3461
q9	8827	8819	8851	8819
q10	3571	3229	3178	3178
q11	591	517	510	510
q12	841	698	639	639
q13	17112	3232	3204	3204
q14	324	276	291	276
q15	531	482	489	482
q16	518	442	447	442
q17	1827	1561	1534	1534
q18	8157	8169	7866	7866
q19	1798	1616	1625	1616
q20	2919	1894	1888	1888
q21	8954	5450	5313	5313
q22	1142	1077	1068	1068
Total cold run time: 78281 ms
Total hot run time: 56534 ms

@doris-robot
Copy link

doris-robot commented Aug 12, 2024

TPC-DS: Total hot run time: 201660 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aec60270acb6995c5dd55134a264548d16f15e38, data reload: false

query1	919	361	359	359
query2	6365	2068	1931	1931
query3	6639	204	214	204
query4	30922	23335	23131	23131
query5	3626	497	517	497
query6	278	166	169	166
query7	4572	308	290	290
query8	245	201	194	194
query9	8511	2497	2471	2471
query10	537	458	461	458
query11	17672	15074	15030	15030
query12	133	106	101	101
query13	1655	366	356	356
query14	9590	6945	7696	6945
query15	259	217	236	217
query16	7604	477	480	477
query17	1302	541	548	541
query18	1937	278	279	278
query19	184	142	145	142
query20	112	104	102	102
query21	208	104	101	101
query22	4550	4337	4307	4307
query23	34489	33636	33634	33634
query24	11020	2731	2672	2672
query25	593	382	391	382
query26	709	161	166	161
query27	2331	284	285	284
query28	6279	2077	2065	2065
query29	796	418	433	418
query30	264	155	146	146
query31	980	759	738	738
query32	99	52	57	52
query33	634	313	278	278
query34	846	472	477	472
query35	913	834	820	820
query36	1096	896	897	896
query37	136	78	77	77
query38	4403	4211	4161	4161
query39	1475	1383	1372	1372
query40	198	116	117	116
query41	46	44	48	44
query42	125	101	99	99
query43	492	467	453	453
query44	1102	734	720	720
query45	234	201	223	201
query46	1105	729	734	729
query47	1848	1759	1749	1749
query48	382	314	296	296
query49	842	432	459	432
query50	795	415	430	415
query51	6702	6645	6758	6645
query52	105	90	100	90
query53	260	188	185	185
query54	897	459	456	456
query55	77	79	76	76
query56	273	261	264	261
query57	1154	1096	1052	1052
query58	236	238	238	238
query59	3204	2703	2760	2703
query60	296	272	270	270
query61	115	112	112	112
query62	786	627	635	627
query63	222	185	186	185
query64	9332	2347	1826	1826
query65	3227	3154	3136	3136
query66	749	336	344	336
query67	15549	14772	14622	14622
query68	4579	541	546	541
query69	407	390	402	390
query70	1160	1144	1145	1144
query71	425	276	278	276
query72	20603	16978	17057	16978
query73	757	329	330	329
query74	9111	8740	8707	8707
query75	4045	2707	2670	2670
query76	2682	990	926	926
query77	692	327	322	322
query78	10698	9227	8866	8866
query79	4469	520	525	520
query80	3119	492	495	492
query81	598	223	219	219
query82	1232	143	128	128
query83	322	166	144	144
query84	265	80	81	80
query85	2070	279	272	272
query86	431	288	297	288
query87	4731	4507	4484	4484
query88	5103	2469	2492	2469
query89	421	282	285	282
query90	2010	197	197	197
query91	124	94	97	94
query92	62	48	47	47
query93	5568	536	527	527
query94	767	257	288	257
query95	350	256	267	256
query96	614	273	275	273
query97	3216	3073	3020	3020
query98	212	194	193	193
query99	1539	1279	1269	1269
Total cold run time: 317618 ms
Total hot run time: 201660 ms

@doris-robot
Copy link

doris-robot commented Aug 12, 2024

ClickBench: Total hot run time: 31.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aec60270acb6995c5dd55134a264548d16f15e38, data reload: false

query1	0.05	0.04	0.05
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.69	0.07	0.06
query5	0.49	0.48	0.48
query6	1.14	0.74	0.72
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.55	0.49	0.48
query10	0.55	0.54	0.54
query11	0.15	0.12	0.12
query12	0.14	0.12	0.12
query13	0.61	0.60	0.59
query14	0.76	0.79	0.79
query15	0.85	0.83	0.83
query16	0.37	0.36	0.36
query17	1.06	0.94	0.97
query18	0.23	0.22	0.22
query19	1.81	1.66	1.73
query20	0.01	0.02	0.01
query21	15.38	0.74	0.65
query22	3.97	6.59	2.70
query23	18.30	1.39	1.30
query24	2.15	0.23	0.22
query25	0.15	0.08	0.08
query26	0.32	0.21	0.21
query27	0.46	0.23	0.22
query28	13.24	1.03	1.01
query29	12.69	3.29	3.25
query30	0.24	0.05	0.05
query31	2.90	0.38	0.38
query32	3.26	0.48	0.47
query33	2.87	2.94	2.97
query34	17.00	4.43	4.36
query35	4.45	4.46	4.43
query36	0.66	0.49	0.49
query37	0.19	0.17	0.16
query38	0.16	0.14	0.15
query39	0.04	0.04	0.04
query40	0.15	0.13	0.13
query41	0.09	0.05	0.05
query42	0.05	0.04	0.04
query43	0.04	0.05	0.04
Total cold run time: 109.59 s
Total hot run time: 31.46 s

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Aug 15, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 15, 2024

TPC-H: Total hot run time: 37270 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit efad5ac20f95459223840bc63dbb5422ea7a9396, data reload: false

------ Round 1 ----------------------------------
q1	17621	4802	4252	4252
q2	2020	180	177	177
q3	11848	1015	1057	1015
q4	10519	763	727	727
q5	7746	2801	2737	2737
q6	221	135	137	135
q7	947	590	590	590
q8	9540	2032	2015	2015
q9	7896	6513	6493	6493
q10	6981	2278	2138	2138
q11	454	235	241	235
q12	391	220	216	216
q13	17834	2978	2954	2954
q14	277	240	240	240
q15	525	491	483	483
q16	490	388	397	388
q17	967	663	719	663
q18	7400	6738	6791	6738
q19	6960	1012	1024	1012
q20	706	332	328	328
q21	3777	2823	2731	2731
q22	1048	1017	1003	1003
Total cold run time: 116168 ms
Total hot run time: 37270 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4500	4252	4271	4252
q2	381	291	269	269
q3	2809	2625	2603	2603
q4	1898	1712	1708	1708
q5	5644	5718	5576	5576
q6	227	139	130	130
q7	2126	1743	1698	1698
q8	3256	3472	3460	3460
q9	8738	8706	8832	8706
q10	3553	3226	3254	3226
q11	604	509	497	497
q12	819	589	617	589
q13	17044	3160	3144	3144
q14	327	290	302	290
q15	530	483	499	483
q16	509	455	467	455
q17	1871	1497	1533	1497
q18	8008	7646	7823	7646
q19	1733	1524	1539	1524
q20	2068	1877	1833	1833
q21	5584	5278	5375	5278
q22	1101	1023	1015	1015
Total cold run time: 73330 ms
Total hot run time: 55879 ms

morningman
morningman reviewed Aug 19, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonSplit.java Outdated

if (split instanceof DataSplit) {
List<DataFileMeta> dataFileMetas = ((DataSplit) split).dataFiles();
this.path = new Path("hdfs://" + dataFileMetas.get(0).fileName());
Copy link
Contributor

@morningman morningman Aug 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why using hdfs://?

Copy link
Contributor Author

@wuwenchi wuwenchi Aug 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it will be checked by location for be.

@morningman
Copy link
Contributor

morningman commented Nov 1, 2024

run buildall

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Nov 1, 2024

run buildall

@morningman morningman force-pushed the improve-external-table-scan branch from 2d434ac to bfcc9b5 Compare November 2, 2024 04:18
@wuwenchi
fix
1ca6e2d 
fix

fix

fix
@wuwenchi
fix
f83781c
@wuwenchi wuwenchi force-pushed the improve-external-table-scan branch from bfcc9b5 to f83781c Compare November 2, 2024 04:26
@wuwenchi
Copy link
Contributor Author

wuwenchi commented Nov 2, 2024

run buildall

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Nov 2, 2024

run buildall

morningman
morningman approved these changes Nov 2, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 2, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2024

PR approved by anyone and no changes requested.

@morningman morningman merged commit 5f07b88 into apache:master Nov 3, 2024
morningman pushed a commit to morningman/doris that referenced this pull request Nov 3, 2024
@wuwenchi @morningman
[improvement](external)add some improvements for external scan (apach…
6027ea2 
…e#38946)

1. add session variable: `use_consistent_hash_for_external_scan`, which
can specify consistent hash for external scan.
2. add session variable: `ignore_split_type`, which can ignore splits of
the specified type, use for performance tuning.
3. add split weight for paimon split with consistent hash.
4. add `executeFilter` for paimon jni split.
@morningman morningman mentioned this pull request Nov 3, 2024
Merged
morningman added a commit that referenced this pull request Nov 4, 2024
@morningman @wuwenchi
[improvement](external)add some improvements for external scan (#38946)…
d1e63c5 
… (#43156)

bp #38946

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
morningman pushed a commit to morningman/doris that referenced this pull request Nov 6, 2024
@wuwenchi @morningman
[improvement](external)add some improvements for external scan (apach…
88fb6de 
…e#38946)

## Proposed changes

1. add session variable: `use_consistent_hash_for_external_scan`, which
can specify consistent hash for external scan.
2. add session variable: `ignore_split_type`, which can ignore splits of
the specified type, use for performance tuning.
3. add split weight for paimon split with consistent hash.
4. add `executeFilter` for paimon jni split.
@morningman morningman mentioned this pull request Nov 6, 2024
Merged
morningman added a commit that referenced this pull request Nov 6, 2024
@morningman @wuwenchi
[improvement](external)add some improvements for external scan #38946 (…
1cc16bf 
…#43310)

cherry pick from #38946

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@zy-kkk zy-kkk zy-kkk approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged doing reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@wuwenchi @doris-robot @morningman @zy-kkk