[opt](file-scanner) add not found file number in profile #37042

morningman · 2024-06-30T14:41:51Z

PR #35319 ignore the not found files in external table by default.
This PR add a BE config ignore_not_found_file_in_external_table to control this behavior,
and the default value is still true.

Also add a new metric NotFoundFileNum, separate from EmptyFileNum,
to record the number of not found files in a query

doris-robot · 2024-06-30T14:41:57Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morningman · 2024-06-30T14:44:16Z

run buildall

github-actions · 2024-06-30T14:47:48Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-06-30T14:50:17Z

clang-tidy review says "All clean, LGTM! 👍"

morningman · 2024-06-30T15:32:06Z

run buildall

github-actions · 2024-06-30T15:38:24Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-06-30T16:21:33Z

TPC-H: Total hot run time: 39960 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5089c7a1e0e313a87d03f12a59485d3173eda471, data reload: false

------ Round 1 ----------------------------------
q1	17616	4421	4304	4304
q2	2021	191	194	191
q3	10506	1238	1168	1168
q4	10197	676	882	676
q5	7549	2661	2649	2649
q6	216	136	140	136
q7	956	597	612	597
q8	9234	2083	2074	2074
q9	9063	6494	6468	6468
q10	9123	3716	3717	3716
q11	454	244	238	238
q12	477	234	233	233
q13	17906	3012	3043	3012
q14	271	221	229	221
q15	526	484	489	484
q16	507	380	376	376
q17	966	655	669	655
q18	8093	7608	7476	7476
q19	7817	1378	1366	1366
q20	701	330	328	328
q21	4879	3255	3999	3255
q22	383	346	337	337
Total cold run time: 119461 ms
Total hot run time: 39960 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4407	4275	4203	4203
q2	368	266	271	266
q3	3076	2923	2931	2923
q4	1984	1800	1758	1758
q5	5528	5497	5496	5496
q6	223	138	146	138
q7	2244	1874	1845	1845
q8	3280	3455	3436	3436
q9	8784	8840	8751	8751
q10	4092	3728	3808	3728
q11	618	492	505	492
q12	833	616	632	616
q13	16460	3159	3151	3151
q14	299	280	275	275
q15	520	505	492	492
q16	470	439	425	425
q17	1845	1541	1481	1481
q18	8333	8031	7718	7718
q19	1753	1655	1671	1655
q20	2123	1881	1881	1881
q21	8724	4714	4782	4714
q22	643	545	538	538
Total cold run time: 76607 ms
Total hot run time: 55982 ms

doris-robot · 2024-06-30T16:32:29Z

TPC-DS: Total hot run time: 171986 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5089c7a1e0e313a87d03f12a59485d3173eda471, data reload: false

query1	921	376	377	376
query2	6449	2289	2303	2289
query3	6640	208	221	208
query4	19577	17411	17072	17072
query5	3590	483	480	480
query6	255	184	181	181
query7	4596	294	297	294
query8	322	295	302	295
query9	8599	2403	2388	2388
query10	561	312	310	310
query11	10530	10117	9973	9973
query12	115	84	82	82
query13	1649	369	360	360
query14	10127	5995	6810	5995
query15	237	181	195	181
query16	7701	270	259	259
query17	1910	554	526	526
query18	1856	266	275	266
query19	193	149	150	149
query20	93	80	81	80
query21	205	131	133	131
query22	4315	4065	3918	3918
query23	33966	33795	33684	33684
query24	10854	3015	2824	2824
query25	644	398	365	365
query26	1401	160	153	153
query27	3062	331	330	330
query28	7770	2136	2121	2121
query29	923	641	619	619
query30	269	166	169	166
query31	1009	787	768	768
query32	95	54	55	54
query33	755	307	280	280
query34	1022	478	479	478
query35	765	636	638	636
query36	1126	976	987	976
query37	165	79	85	79
query38	2965	2897	2839	2839
query39	880	820	854	820
query40	207	130	129	129
query41	55	57	50	50
query42	110	106	107	106
query43	611	551	565	551
query44	1203	726	731	726
query45	189	160	172	160
query46	1084	732	714	714
query47	1865	1769	1783	1769
query48	369	292	289	289
query49	850	412	418	412
query50	774	387	397	387
query51	6830	6723	6861	6723
query52	99	97	100	97
query53	359	290	296	290
query54	883	453	431	431
query55	74	74	74	74
query56	285	268	265	265
query57	1145	1022	1066	1022
query58	250	229	245	229
query59	3400	3170	3033	3033
query60	285	275	307	275
query61	95	95	89	89
query62	596	444	442	442
query63	322	285	287	285
query64	8844	2234	1727	1727
query65	3201	3103	3070	3070
query66	756	319	318	318
query67	15357	14900	14852	14852
query68	4866	531	521	521
query69	561	492	339	339
query70	1106	1163	1145	1145
query71	414	279	279	279
query72	6999	5515	5166	5166
query73	746	322	321	321
query74	6006	5589	5461	5461
query75	3505	2639	2689	2639
query76	3106	945	928	928
query77	702	303	303	303
query78	10254	9841	9764	9764
query79	2190	525	501	501
query80	925	472	472	472
query81	545	222	221	221
query82	1051	106	107	106
query83	256	168	173	168
query84	231	87	85	85
query85	1120	287	274	274
query86	448	337	305	305
query87	3290	3068	3104	3068
query88	3507	2402	2353	2353
query89	469	369	388	369
query90	1689	187	191	187
query91	124	100	98	98
query92	59	50	53	50
query93	2008	511	495	495
query94	915	246	181	181
query95	402	320	316	316
query96	583	265	280	265
query97	3188	3046	3099	3046
query98	220	196	211	196
query99	1047	853	844	844
Total cold run time: 270742 ms
Total hot run time: 171986 ms

doris-robot · 2024-06-30T16:38:00Z

ClickBench: Total hot run time: 30.09 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5089c7a1e0e313a87d03f12a59485d3173eda471, data reload: false

query1	0.04	0.03	0.02
query2	0.08	0.04	0.05
query3	0.23	0.06	0.05
query4	1.67	0.07	0.06
query5	0.50	0.47	0.49
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.55	0.54
query11	0.15	0.11	0.12
query12	0.15	0.11	0.12
query13	0.60	0.60	0.60
query14	0.79	0.78	0.79
query15	0.85	0.81	0.81
query16	0.37	0.34	0.36
query17	0.99	1.03	0.99
query18	0.23	0.25	0.26
query19	1.77	1.69	1.75
query20	0.01	0.01	0.01
query21	15.42	0.72	0.64
query22	3.76	8.66	1.46
query23	18.28	1.32	1.32
query24	2.11	0.23	0.22
query25	0.16	0.09	0.09
query26	0.27	0.18	0.17
query27	0.08	0.07	0.08
query28	13.22	1.04	1.01
query29	12.66	3.26	3.28
query30	0.25	0.06	0.06
query31	2.86	0.38	0.39
query32	3.26	0.47	0.49
query33	2.85	2.96	2.88
query34	17.10	4.38	4.38
query35	4.51	4.51	4.47
query36	0.65	0.46	0.46
query37	0.19	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.04	0.04
query40	0.17	0.15	0.16
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.05
Total cold run time: 108.92 s
Total hot run time: 30.09 s

AshinGau · 2024-07-01T06:59:54Z

LGTM

github-actions · 2024-07-01T07:00:16Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-01T07:00:18Z

PR approved by anyone and no changes requested.

morningman · 2024-07-01T07:35:02Z

run buildall

github-actions · 2024-07-01T07:41:01Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-01T08:49:36Z

TPC-H: Total hot run time: 40490 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0d0c5b1e2fe293e0cabc2f32d56d0be5f8d4e15f, data reload: false

------ Round 1 ----------------------------------
q1	18911	4544	4441	4441
q2	2664	203	198	198
q3	11598	1211	1180	1180
q4	10314	853	889	853
q5	7602	2778	2671	2671
q6	220	140	142	140
q7	968	607	616	607
q8	9230	2104	2076	2076
q9	9025	6515	6508	6508
q10	8950	3763	3754	3754
q11	471	248	244	244
q12	399	239	234	234
q13	17848	2992	2974	2974
q14	276	229	226	226
q15	544	477	483	477
q16	517	380	377	377
q17	991	721	689	689
q18	8163	7512	7468	7468
q19	9245	1567	1484	1484
q20	687	337	326	326
q21	5007	3224	3295	3224
q22	402	347	339	339
Total cold run time: 124032 ms
Total hot run time: 40490 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4436	4275	4235	4235
q2	410	263	260	260
q3	2964	2777	2696	2696
q4	1928	1598	1612	1598
q5	5281	5360	5297	5297
q6	216	136	133	133
q7	2137	1694	1758	1694
q8	3228	3376	3334	3334
q9	8401	8365	8373	8365
q10	3902	3684	3701	3684
q11	566	492	494	492
q12	798	584	621	584
q13	17559	3036	3018	3018
q14	285	269	251	251
q15	514	482	475	475
q16	471	412	423	412
q17	1793	1515	1492	1492
q18	7659	7464	7446	7446
q19	1705	1637	1669	1637
q20	2027	1808	1807	1807
q21	4896	4704	4715	4704
q22	625	548	555	548
Total cold run time: 71801 ms
Total hot run time: 54162 ms

doris-robot · 2024-07-01T09:00:37Z

TPC-DS: Total hot run time: 172863 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0d0c5b1e2fe293e0cabc2f32d56d0be5f8d4e15f, data reload: false

query1	913	389	384	384
query2	6471	2351	2315	2315
query3	6653	211	219	211
query4	19415	17562	17143	17143
query5	4178	485	468	468
query6	275	171	162	162
query7	4606	307	291	291
query8	328	290	285	285
query9	8688	2395	2390	2390
query10	612	316	298	298
query11	10434	9902	10160	9902
query12	131	87	82	82
query13	1642	370	374	370
query14	8768	8063	7197	7197
query15	230	195	194	194
query16	7824	282	306	282
query17	1854	576	515	515
query18	1952	275	273	273
query19	194	149	163	149
query20	93	82	87	82
query21	211	129	123	123
query22	4185	4123	4018	4018
query23	33848	33104	33083	33083
query24	11483	2761	2852	2761
query25	678	358	360	358
query26	1786	157	156	156
query27	2946	327	313	313
query28	7607	2068	2057	2057
query29	1053	618	609	609
query30	273	154	152	152
query31	951	734	748	734
query32	89	52	58	52
query33	767	284	291	284
query34	981	471	473	471
query35	736	649	611	611
query36	1101	947	963	947
query37	162	74	81	74
query38	2878	2755	2761	2755
query39	837	814	799	799
query40	282	129	127	127
query41	58	51	55	51
query42	120	97	104	97
query43	583	532	569	532
query44	1262	724	725	724
query45	190	159	159	159
query46	1078	709	737	709
query47	1835	1771	1755	1755
query48	371	300	298	298
query49	1178	422	419	419
query50	770	388	396	388
query51	6907	6824	6766	6766
query52	111	102	94	94
query53	363	294	303	294
query54	997	445	438	438
query55	76	75	75	75
query56	289	275	281	275
query57	1141	1074	1034	1034
query58	254	232	249	232
query59	3291	3289	3211	3211
query60	306	315	288	288
query61	95	95	93	93
query62	662	432	470	432
query63	316	292	295	292
query64	9922	2249	1762	1762
query65	3165	3094	3095	3094
query66	1433	340	330	330
query67	15353	15078	14928	14928
query68	4657	544	549	544
query69	470	314	324	314
query70	1156	1148	1135	1135
query71	398	279	278	278
query72	7396	5244	5697	5244
query73	749	323	322	322
query74	5989	5528	5657	5528
query75	3400	2717	2692	2692
query76	2854	934	896	896
query77	464	310	308	308
query78	10221	10006	9693	9693
query79	2557	526	527	526
query80	1094	463	475	463
query81	546	218	223	218
query82	1038	111	107	107
query83	250	171	182	171
query84	243	87	86	86
query85	1389	293	269	269
query86	467	323	330	323
query87	3281	3125	3115	3115
query88	3610	2385	2358	2358
query89	479	383	377	377
query90	1805	187	188	187
query91	131	99	102	99
query92	73	49	54	49
query93	1924	510	510	510
query94	1288	198	187	187
query95	405	317	320	317
query96	598	263	267	263
query97	3274	2984	3018	2984
query98	219	210	200	200
query99	1138	861	853	853
Total cold run time: 273804 ms
Total hot run time: 172863 ms

doris-robot · 2024-07-01T09:06:07Z

ClickBench: Total hot run time: 31.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0d0c5b1e2fe293e0cabc2f32d56d0be5f8d4e15f, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.10	0.09
query5	0.49	0.48	0.49
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.55	0.49	0.50
query10	0.55	0.54	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.60	0.60
query14	0.81	0.78	0.77
query15	0.84	0.83	0.82
query16	0.37	0.36	0.37
query17	1.07	1.06	1.04
query18	0.23	0.26	0.23
query19	1.92	1.76	1.83
query20	0.02	0.01	0.01
query21	15.45	0.76	0.66
query22	4.13	6.70	2.21
query23	18.32	1.39	1.29
query24	2.12	0.22	0.22
query25	0.16	0.09	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.07
query28	13.22	1.02	0.99
query29	12.64	3.35	3.29
query30	0.25	0.07	0.06
query31	2.86	0.40	0.39
query32	3.27	0.48	0.48
query33	2.92	2.92	2.93
query34	17.05	4.38	4.45
query35	4.48	4.47	4.46
query36	0.65	0.48	0.48
query37	0.18	0.16	0.17
query38	0.15	0.16	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.15
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.56 s
Total hot run time: 31.11 s

kaka11chen

LGTM

github-actions · 2024-07-02T03:22:08Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-03T02:57:15Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-03T04:12:29Z

TPC-H: Total hot run time: 40181 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9cc25a64da3e706c50ac95225e1556631ebfaa74, data reload: false

------ Round 1 ----------------------------------
q1	17974	4570	4428	4428
q2	2297	194	188	188
q3	10486	1249	1168	1168
q4	10205	781	790	781
q5	7526	2657	2683	2657
q6	222	137	139	137
q7	974	591	602	591
q8	9225	2108	2065	2065
q9	8795	6504	6488	6488
q10	9000	3739	3712	3712
q11	479	238	239	238
q12	407	240	225	225
q13	17836	3008	3006	3006
q14	263	222	225	222
q15	527	491	485	485
q16	518	378	379	378
q17	970	660	745	660
q18	8146	7468	7558	7468
q19	5817	1479	1461	1461
q20	672	327	327	327
q21	5034	3165	3856	3165
q22	414	331	333	331
Total cold run time: 117787 ms
Total hot run time: 40181 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4369	4302	4271	4271
q2	385	271	268	268
q3	2991	2742	2750	2742
q4	1844	1609	1580	1580
q5	5303	5315	5296	5296
q6	223	134	131	131
q7	2135	1766	1750	1750
q8	3184	3337	3358	3337
q9	8313	8331	8359	8331
q10	3897	3693	3702	3693
q11	571	507	502	502
q12	765	596	629	596
q13	17607	3009	2959	2959
q14	302	272	265	265
q15	516	495	490	490
q16	477	419	436	419
q17	1825	1479	1463	1463
q18	7781	7623	7431	7431
q19	2272	1601	1520	1520
q20	1989	1800	1785	1785
q21	4872	4891	4703	4703
q22	622	525	542	525
Total cold run time: 72243 ms
Total hot run time: 54057 ms

doris-robot · 2024-07-03T04:23:30Z

TPC-DS: Total hot run time: 172313 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9cc25a64da3e706c50ac95225e1556631ebfaa74, data reload: false

query1	932	390	376	376
query2	6449	2406	2588	2406
query3	6658	208	209	208
query4	19532	17428	17336	17336
query5	4175	485	487	485
query6	257	178	162	162
query7	4604	295	299	295
query8	339	294	292	292
query9	8498	2484	2468	2468
query10	602	281	276	276
query11	10457	10045	9980	9980
query12	136	87	85	85
query13	1635	356	359	356
query14	9038	7043	6166	6166
query15	227	188	186	186
query16	7557	275	264	264
query17	1408	535	510	510
query18	1926	271	270	270
query19	191	147	152	147
query20	87	82	89	82
query21	234	132	132	132
query22	4274	4115	4000	4000
query23	33687	32965	33213	32965
query24	12074	2792	2846	2792
query25	669	360	385	360
query26	1813	164	158	158
query27	3088	312	318	312
query28	7412	2155	2123	2123
query29	1132	632	607	607
query30	288	158	154	154
query31	948	745	727	727
query32	90	52	55	52
query33	771	282	305	282
query34	965	459	472	459
query35	739	641	610	610
query36	1085	959	944	944
query37	210	76	78	76
query38	2848	2774	2744	2744
query39	892	801	779	779
query40	279	124	120	120
query41	54	53	54	53
query42	125	94	100	94
query43	623	548	576	548
query44	1192	726	728	726
query45	189	165	164	164
query46	1068	714	714	714
query47	1860	1747	1770	1747
query48	358	302	295	295
query49	1218	409	411	409
query50	763	390	387	387
query51	6876	6801	6707	6707
query52	103	91	100	91
query53	360	293	286	286
query54	983	447	440	440
query55	75	75	74	74
query56	285	259	265	259
query57	1144	1031	1030	1030
query58	248	255	257	255
query59	3671	3136	3186	3136
query60	307	276	287	276
query61	93	92	93	92
query62	637	438	442	438
query63	323	290	285	285
query64	9925	2239	1849	1849
query65	3187	3119	3157	3119
query66	1415	355	345	345
query67	15706	15174	14949	14949
query68	4702	541	563	541
query69	466	298	314	298
query70	1174	1150	1153	1150
query71	396	272	274	272
query72	7269	5540	5432	5432
query73	748	330	322	322
query74	5912	5474	5497	5474
query75	3385	2666	2678	2666
query76	2505	1027	943	943
query77	437	293	301	293
query78	10289	9855	9760	9760
query79	2304	519	522	519
query80	1024	469	479	469
query81	583	221	220	220
query82	740	110	107	107
query83	250	162	167	162
query84	242	86	81	81
query85	2025	280	278	278
query86	494	316	324	316
query87	3229	3098	3061	3061
query88	4172	2462	2454	2454
query89	474	377	375	375
query90	1841	183	182	182
query91	125	101	99	99
query92	59	49	50	49
query93	2439	518	511	511
query94	1257	191	189	189
query95	401	318	304	304
query96	606	266	268	266
query97	3179	3021	2988	2988
query98	216	194	198	194
query99	1269	824	827	824
Total cold run time: 275206 ms
Total hot run time: 172313 ms

doris-robot · 2024-07-03T04:29:01Z

ClickBench: Total hot run time: 30.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9cc25a64da3e706c50ac95225e1556631ebfaa74, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.50	0.49
query6	1.13	0.74	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.56	0.49	0.48
query10	0.54	0.54	0.54
query11	0.17	0.12	0.11
query12	0.15	0.12	0.13
query13	0.60	0.58	0.60
query14	0.77	0.80	0.79
query15	0.83	0.80	0.81
query16	0.36	0.35	0.35
query17	0.97	1.01	0.98
query18	0.21	0.28	0.24
query19	1.91	1.73	1.75
query20	0.01	0.01	0.01
query21	15.45	0.74	0.66
query22	4.34	7.48	1.81
query23	18.32	1.30	1.24
query24	2.13	0.23	0.22
query25	0.16	0.09	0.09
query26	0.26	0.17	0.18
query27	0.08	0.07	0.09
query28	13.19	1.02	1.00
query29	12.63	3.29	3.30
query30	0.27	0.06	0.05
query31	2.89	0.38	0.38
query32	3.27	0.48	0.47
query33	2.85	2.90	2.88
query34	17.31	4.40	4.44
query35	4.49	4.45	4.49
query36	0.66	0.47	0.48
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.03
query40	0.18	0.15	0.14
query41	0.10	0.04	0.05
query42	0.05	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.86 s
Total hot run time: 30.44 s

morningman · 2024-07-04T12:12:11Z

run buildall

github-actions · 2024-07-04T12:18:20Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-04T13:31:35Z

TPC-H: Total hot run time: 40205 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f1ceea8f675eca3275b865f52b04796a81ae0c2, data reload: false

------ Round 1 ----------------------------------
q1	17867	4570	4382	4382
q2	2483	207	200	200
q3	10886	1285	1193	1193
q4	11752	811	824	811
q5	7530	2712	2671	2671
q6	227	140	136	136
q7	976	605	626	605
q8	9340	2116	2072	2072
q9	8808	6527	6498	6498
q10	8892	3678	3767	3678
q11	470	230	233	230
q12	439	239	219	219
q13	17764	2977	2978	2977
q14	261	225	215	215
q15	529	482	487	482
q16	525	385	372	372
q17	975	698	723	698
q18	8065	7466	7531	7466
q19	6693	1536	1554	1536
q20	660	326	312	312
q21	4924	3115	3291	3115
q22	386	340	337	337
Total cold run time: 120452 ms
Total hot run time: 40205 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4304	4240	4176	4176
q2	384	269	258	258
q3	2998	2782	2766	2766
q4	1841	1625	1590	1590
q5	5290	5322	5291	5291
q6	218	130	128	128
q7	2127	1714	1754	1714
q8	3191	3337	3290	3290
q9	8393	8338	8346	8338
q10	3858	3738	3630	3630
q11	589	488	477	477
q12	769	643	610	610
q13	16865	2972	2973	2972
q14	295	289	263	263
q15	512	479	478	478
q16	475	416	410	410
q17	1777	1450	1475	1450
q18	7693	7508	7448	7448
q19	1682	1638	1574	1574
q20	1970	1774	1774	1774
q21	8924	4727	4763	4727
q22	611	560	552	552
Total cold run time: 74766 ms
Total hot run time: 53916 ms

doris-robot · 2024-07-04T13:42:57Z

TPC-DS: Total hot run time: 171886 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f1ceea8f675eca3275b865f52b04796a81ae0c2, data reload: false

query1	923	383	376	376
query2	6454	2468	2471	2468
query3	6650	205	210	205
query4	19602	17466	17386	17386
query5	4180	487	479	479
query6	296	184	166	166
query7	4597	299	288	288
query8	329	300	297	297
query9	8523	2389	2370	2370
query10	618	323	285	285
query11	10512	10024	10124	10024
query12	138	93	85	85
query13	1654	362	370	362
query14	10143	7231	6479	6479
query15	226	184	190	184
query16	7809	317	315	315
query17	1821	552	544	544
query18	1924	282	278	278
query19	209	161	152	152
query20	93	81	84	81
query21	210	129	127	127
query22	4274	3977	4054	3977
query23	33687	32955	33281	32955
query24	11493	2872	2745	2745
query25	689	380	388	380
query26	1646	156	152	152
query27	2946	313	314	313
query28	7407	2066	2045	2045
query29	1059	636	622	622
query30	286	157	158	157
query31	955	747	739	739
query32	98	58	57	57
query33	794	317	302	302
query34	897	457	463	457
query35	759	634	631	631
query36	1128	927	905	905
query37	148	76	76	76
query38	2875	2713	2771	2713
query39	856	813	816	813
query40	291	127	127	127
query41	57	54	56	54
query42	120	98	100	98
query43	627	540	579	540
query44	1175	722	723	722
query45	195	165	170	165
query46	1071	707	725	707
query47	1896	1773	1786	1773
query48	368	301	300	300
query49	1162	419	420	419
query50	771	382	390	382
query51	6941	6824	6787	6787
query52	105	93	95	93
query53	355	289	296	289
query54	1009	456	441	441
query55	74	74	74	74
query56	303	280	282	280
query57	1156	1040	1035	1035
query58	276	375	263	263
query59	3594	3472	3460	3460
query60	304	277	286	277
query61	98	97	90	90
query62	680	453	460	453
query63	325	293	283	283
query64	9891	2293	1771	1771
query65	3193	3103	3112	3103
query66	1351	354	330	330
query67	15916	15004	15164	15004
query68	8338	540	557	540
query69	739	426	344	344
query70	1181	1114	1077	1077
query71	523	270	276	270
query72	9037	5384	5545	5384
query73	816	317	316	316
query74	5956	5535	5490	5490
query75	5120	2686	2652	2652
query76	5095	897	863	863
query77	778	294	302	294
query78	9563	8923	8823	8823
query79	9769	512	510	510
query80	1019	486	460	460
query81	557	227	215	215
query82	362	106	105	105
query83	337	164	162	162
query84	273	83	84	83
query85	1017	314	274	274
query86	357	304	305	304
query87	3286	3062	3113	3062
query88	4112	2361	2364	2361
query89	518	381	385	381
query90	2116	187	180	180
query91	131	100	99	99
query92	60	49	47	47
query93	6974	494	491	491
query94	1419	213	206	206
query95	405	305	304	304
query96	615	265	257	257
query97	3213	2998	3031	2998
query98	225	201	191	191
query99	1137	848	833	833
Total cold run time: 297260 ms
Total hot run time: 171886 ms

doris-robot · 2024-07-04T13:48:28Z

ClickBench: Total hot run time: 30.47 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f1ceea8f675eca3275b865f52b04796a81ae0c2, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.69	0.08	0.07
query5	0.47	0.48	0.48
query6	1.14	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.55	0.55	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.76	0.76	0.78
query15	0.85	0.81	0.82
query16	0.36	0.35	0.34
query17	1.00	1.00	1.04
query18	0.25	0.25	0.24
query19	1.84	1.79	1.69
query20	0.02	0.01	0.01
query21	15.39	0.77	0.64
query22	4.16	7.55	1.92
query23	18.33	1.30	1.25
query24	2.20	0.22	0.21
query25	0.15	0.08	0.08
query26	0.31	0.22	0.20
query27	0.45	0.23	0.22
query28	13.25	1.02	0.99
query29	12.62	3.27	3.22
query30	0.25	0.06	0.06
query31	2.86	0.38	0.40
query32	3.28	0.48	0.48
query33	2.91	2.89	2.96
query34	17.09	4.35	4.33
query35	4.40	4.46	4.44
query36	0.65	0.47	0.46
query37	0.17	0.15	0.17
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.16	0.12	0.12
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.04	0.05	0.03
Total cold run time: 109.82 s
Total hot run time: 30.47 s

kaka11chen

LGTM

github-actions · 2024-07-05T04:34:37Z

PR approved by at least one committer and no changes requested.

PR apache#35319 ignore the not found files in external table by default. This PR add a BE config `ignore_not_found_file_in_external_table` to control this behavior, and the default value is still `true`. Also add a new metric `NotFoundFileNum`, separate from `EmptyFileNum`, to record the number of not found files in a query

…7764) bp #37042

PR #35319 ignore the not found files in external table by default. This PR add a BE config `ignore_not_found_file_in_external_table` to control this behavior, and the default value is still `true`. Also add a new metric `NotFoundFileNum`, separate from `EmptyFileNum`, to record the number of not found files in a query

followup #37042 1. The previous PR miss one place to record "not found" file num 2. Catch "NoSuchKey" error and treat it as "not found"

followup apache#37042 1. The previous PR miss one place to record "not found" file num 2. Catch "NoSuchKey" error and treat it as "not found"

followup #37042 1. The previous PR miss one place to record "not found" file num 2. Catch "NoSuchKey" error and treat it as "not found"

morningman added dev/2.1.4-merged dev/2.1.x and removed dev/2.1.4-merged labels Jun 30, 2024

AshinGau previously approved these changes Jul 1, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 1, 2024

wuwenchi approved these changes Jul 1, 2024

View reviewed changes

morningman dismissed AshinGau’s stale review via 0d0c5b1 July 1, 2024 07:34

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 1, 2024

wuwenchi approved these changes Jul 2, 2024

View reviewed changes

kaka11chen previously approved these changes Jul 2, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 2, 2024

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 3, 2024

morningman added 3 commits July 4, 2024 20:11

[opt](file-scanner) add not found file number in profile

04d1dc4

format

03393e5

format

6f1ceea

morningman force-pushed the not_found_number branch from 9cc25a6 to 6f1ceea Compare July 4, 2024 12:12

kaka11chen approved these changes Jul 5, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 5, 2024

morningman merged commit 71a23d0 into apache:master Jul 5, 2024

morningman mentioned this pull request Jul 14, 2024

[opt](file-scanner) add not found file number in profile (#37042) #37764

Merged

morningman added a commit that referenced this pull request Jul 15, 2024

[opt](file-scanner) add not found file number in profile (#37042) (#3…

a4d37d9

…7764) bp #37042

morningman added dev/2.1.5-merged and removed dev/2.1.x labels Jul 15, 2024

morningman mentioned this pull request Jul 23, 2024

[fix](external) record not found file number #38253

Merged

morningman added a commit that referenced this pull request Jul 24, 2024

[fix](external) record not found file number (#38253)

124de2a

followup #37042 1. The previous PR miss one place to record "not found" file num 2. Catch "NoSuchKey" error and treat it as "not found"

dataroaring pushed a commit that referenced this pull request Jul 24, 2024

[fix](external) record not found file number (#38253)

4023fb8

followup #37042 1. The previous PR miss one place to record "not found" file num 2. Catch "NoSuchKey" error and treat it as "not found"

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Closed

[opt](file-scanner) add not found file number in profile #37042

[opt](file-scanner) add not found file number in profile #37042

Uh oh!

Conversation

morningman commented Jun 30, 2024

Uh oh!

doris-robot commented Jun 30, 2024

Uh oh!

morningman commented Jun 30, 2024

Uh oh!

github-actions bot commented Jun 30, 2024

Uh oh!

github-actions bot commented Jun 30, 2024

Uh oh!

morningman commented Jun 30, 2024

Uh oh!

github-actions bot commented Jun 30, 2024

Uh oh!

doris-robot commented Jun 30, 2024

Uh oh!

doris-robot commented Jun 30, 2024

Uh oh!

doris-robot commented Jun 30, 2024

Uh oh!

AshinGau commented Jul 1, 2024

Uh oh!

github-actions bot commented Jul 1, 2024

Uh oh!

github-actions bot commented Jul 1, 2024

Uh oh!

morningman commented Jul 1, 2024

Uh oh!

github-actions bot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

kaka11chen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 2, 2024

Uh oh!

github-actions bot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

morningman commented Jul 4, 2024

Uh oh!

github-actions bot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

kaka11chen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants