Skip to content

[opt](nereids) catch all exceptions in StatsCalculator #49415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
englefly merged 4 commits into from
Apr 3, 2025
Merged

[opt](nereids) catch all exceptions in StatsCalculator #49415

englefly merged 4 commits into from
Apr 3, 2025

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Mar 24, 2025

What problem does this PR solve?

catch all exceptions in ExpressionEstimation so that any error in ExpressionEstimation does not break the query.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@englefly
Copy link
Contributor Author

englefly commented Mar 24, 2025

run buildall

@924060929 924060929 requested a review from yujun777 March 24, 2025 10:20
@doris-robot
Copy link

doris-robot commented Mar 24, 2025

TPC-H: Total hot run time: 34723 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c9cd93acb7b17b3b11fba103b0c69ffa49680039, data reload: false

------ Round 1 ----------------------------------
q1	26180	5091	5058	5058
q2	2068	299	171	171
q3	10769	1256	726	726
q4	10327	1042	554	554
q5	9572	2476	2406	2406
q6	267	167	136	136
q7	945	795	623	623
q8	9331	1426	1151	1151
q9	7577	5172	5127	5127
q10	6914	2315	1894	1894
q11	492	294	259	259
q12	367	364	222	222
q13	17777	3706	3138	3138
q14	233	231	214	214
q15	542	486	484	484
q16	630	613	597	597
q17	595	884	369	369
q18	7530	7268	7327	7268
q19	1533	978	581	581
q20	338	342	201	201
q21	4150	3664	2593	2593
q22	1059	1018	951	951
Total cold run time: 119196 ms
Total hot run time: 34723 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5308	5173	5355	5173
q2	241	330	230	230
q3	2188	2653	2469	2469
q4	1529	1981	1574	1574
q5	4567	4400	4429	4400
q6	220	168	132	132
q7	1987	1936	1818	1818
q8	2647	2650	2531	2531
q9	7360	7097	7159	7097
q10	3025	3233	2777	2777
q11	589	506	521	506
q12	713	783	603	603
q13	3525	4144	3325	3325
q14	285	305	268	268
q15	531	485	470	470
q16	652	673	665	665
q17	1170	1545	1410	1410
q18	8091	7685	7512	7512
q19	810	795	1002	795
q20	2011	2035	1879	1879
q21	5589	5035	4876	4876
q22	1120	1069	1021	1021
Total cold run time: 54158 ms
Total hot run time: 51531 ms

@doris-robot
Copy link

doris-robot commented Mar 24, 2025

TPC-DS: Total hot run time: 193490 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c9cd93acb7b17b3b11fba103b0c69ffa49680039, data reload: false

query1	1416	1077	1060	1060
query2	6276	1994	1974	1974
query3	11020	4548	4601	4548
query4	25948	23803	23028	23028
query5	4817	670	504	504
query6	325	209	211	209
query7	4001	505	293	293
query8	314	264	247	247
query9	8496	2617	2604	2604
query10	504	329	274	274
query11	15527	15150	14814	14814
query12	162	108	105	105
query13	1553	544	396	396
query14	9357	6263	6195	6195
query15	208	185	172	172
query16	7598	646	493	493
query17	1116	730	609	609
query18	2044	403	323	323
query19	204	197	181	181
query20	128	135	119	119
query21	212	126	103	103
query22	4597	4516	4436	4436
query23	34660	33687	33273	33273
query24	8449	2498	2493	2493
query25	501	476	413	413
query26	1195	270	145	145
query27	2811	495	335	335
query28	5005	2445	2429	2429
query29	713	562	447	447
query30	280	228	197	197
query31	972	884	806	806
query32	74	66	62	62
query33	580	379	317	317
query34	927	893	508	508
query35	809	836	777	777
query36	979	1073	919	919
query37	128	99	79	79
query38	4217	4192	4127	4127
query39	1487	1444	1458	1444
query40	208	123	105	105
query41	55	55	52	52
query42	127	107	109	107
query43	526	517	502	502
query44	1367	822	836	822
query45	180	179	174	174
query46	885	1059	665	665
query47	1812	1899	1809	1809
query48	397	438	309	309
query49	797	515	432	432
query50	739	770	432	432
query51	4347	4312	4296	4296
query52	114	115	108	108
query53	236	264	184	184
query54	515	530	434	434
query55	83	81	84	81
query56	291	304	251	251
query57	1205	1267	1129	1129
query58	257	249	248	248
query59	2770	2793	2756	2756
query60	307	302	276	276
query61	159	169	166	166
query62	786	753	691	691
query63	240	195	197	195
query64	4052	1089	682	682
query65	4458	4334	4345	4334
query66	996	402	297	297
query67	15948	15573	15562	15562
query68	9585	891	507	507
query69	495	303	270	270
query70	1231	1117	1130	1117
query71	480	308	276	276
query72	5646	5026	5029	5026
query73	685	584	346	346
query74	9032	9051	8724	8724
query75	4358	3286	2693	2693
query76	4019	1197	766	766
query77	991	384	281	281
query78	10085	10161	9259	9259
query79	2367	806	565	565
query80	627	515	442	442
query81	509	260	226	226
query82	480	122	93	93
query83	207	172	155	155
query84	297	99	75	75
query85	797	366	312	312
query86	385	307	284	284
query87	4496	4671	4454	4454
query88	3551	2237	2240	2237
query89	401	318	276	276
query90	1905	218	221	218
query91	144	147	108	108
query92	73	59	56	56
query93	1899	1058	577	577
query94	672	427	305	305
query95	366	278	278	278
query96	492	574	272	272
query97	3337	3416	3248	3248
query98	223	204	194	194
query99	1450	1411	1293	1293
Total cold run time: 284752 ms
Total hot run time: 193490 ms

morrySnow
morrySnow reviewed Mar 24, 2025
View reviewed changes
Copy link
Contributor

@morrySnow morrySnow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should also process

  • FilterEstimation
  • JoinEstimation

@doris-robot
Copy link

doris-robot commented Mar 24, 2025

ClickBench: Total hot run time: 31.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c9cd93acb7b17b3b11fba103b0c69ffa49680039, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.11	0.10
query3	0.26	0.20	0.20
query4	1.59	0.20	0.18
query5	0.60	0.59	0.61
query6	1.20	0.72	0.73
query7	0.03	0.02	0.01
query8	0.04	0.04	0.04
query9	0.59	0.52	0.53
query10	0.56	0.60	0.56
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.60	0.60
query14	2.80	2.70	2.69
query15	0.95	0.85	0.86
query16	0.37	0.37	0.38
query17	1.05	1.04	1.03
query18	0.22	0.20	0.20
query19	1.96	2.02	1.81
query20	0.01	0.01	0.02
query21	15.36	0.91	0.55
query22	0.76	1.11	0.64
query23	15.07	1.41	0.59
query24	7.04	1.66	0.87
query25	0.51	0.17	0.08
query26	0.64	0.18	0.13
query27	0.05	0.04	0.05
query28	9.92	0.92	0.42
query29	12.56	3.92	3.28
query30	0.26	0.09	0.07
query31	2.83	0.60	0.39
query32	3.25	0.55	0.46
query33	3.04	3.10	3.08
query34	15.67	5.09	4.54
query35	4.49	4.52	4.54
query36	0.67	0.49	0.49
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.93 s
Total hot run time: 31.21 s

@englefly
Copy link
Contributor Author

englefly commented Mar 24, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 25, 2025

TPC-H: Total hot run time: 34695 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2fd4fd2f9e461390767043b6c2a72b07358a54a9, data reload: false

------ Round 1 ----------------------------------
q1	25481	5047	5030	5030
q2	2057	302	164	164
q3	10391	1236	668	668
q4	10227	1004	552	552
q5	7527	2445	2312	2312
q6	187	165	134	134
q7	914	766	604	604
q8	9308	1289	1067	1067
q9	6805	5237	5095	5095
q10	6862	2310	1907	1907
q11	478	274	254	254
q12	346	362	221	221
q13	17767	3731	3069	3069
q14	220	228	210	210
q15	541	492	498	492
q16	617	612	595	595
q17	596	858	341	341
q18	7603	7298	7106	7106
q19	1437	938	549	549
q20	327	331	197	197
q21	3900	3369	3136	3136
q22	1062	1037	992	992
Total cold run time: 114653 ms
Total hot run time: 34695 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5328	5099	5119	5099
q2	235	325	240	240
q3	2188	2664	2296	2296
q4	1425	1881	1465	1465
q5	4468	4391	4442	4391
q6	215	173	129	129
q7	2023	1948	1757	1757
q8	2588	2590	2622	2590
q9	7352	7172	7044	7044
q10	2979	3203	2728	2728
q11	600	516	502	502
q12	678	761	641	641
q13	3467	3922	3432	3432
q14	315	315	266	266
q15	524	487	471	471
q16	654	678	642	642
q17	1164	1535	1398	1398
q18	7800	7571	7477	7477
q19	817	798	884	798
q20	1955	1981	1903	1903
q21	5417	5045	4915	4915
q22	1067	1068	1027	1027
Total cold run time: 53259 ms
Total hot run time: 51211 ms

@doris-robot
Copy link

doris-robot commented Mar 25, 2025

TPC-DS: Total hot run time: 194532 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2fd4fd2f9e461390767043b6c2a72b07358a54a9, data reload: false

query1	1415	1048	1065	1048
query2	6215	1970	2005	1970
query3	11171	4662	4683	4662
query4	26011	23753	23583	23583
query5	4968	656	490	490
query6	308	197	192	192
query7	3995	488	272	272
query8	312	248	240	240
query9	8520	2607	2610	2607
query10	492	314	256	256
query11	15612	15100	14892	14892
query12	164	113	108	108
query13	1562	529	404	404
query14	9156	6244	6296	6244
query15	199	195	173	173
query16	7625	651	496	496
query17	1103	758	622	622
query18	2070	432	355	355
query19	218	197	219	197
query20	125	121	123	121
query21	205	125	102	102
query22	4443	4709	4356	4356
query23	34444	33602	33668	33602
query24	7603	2443	2492	2443
query25	511	467	393	393
query26	736	268	146	146
query27	2720	504	328	328
query28	3133	2435	2463	2435
query29	644	579	444	444
query30	289	224	194	194
query31	906	862	803	803
query32	70	67	63	63
query33	510	391	323	323
query34	751	866	510	510
query35	805	888	791	791
query36	968	1020	915	915
query37	129	102	77	77
query38	4182	4167	4182	4167
query39	1510	1440	1487	1440
query40	213	121	107	107
query41	53	53	52	52
query42	123	108	108	108
query43	505	528	485	485
query44	1367	826	821	821
query45	189	180	176	176
query46	866	1041	631	631
query47	1828	1872	1802	1802
query48	381	421	315	315
query49	699	522	434	434
query50	723	765	442	442
query51	4324	4244	4238	4238
query52	113	111	101	101
query53	231	266	184	184
query54	493	503	420	420
query55	88	84	88	84
query56	276	289	281	281
query57	1189	1193	1135	1135
query58	292	249	247	247
query59	2761	2952	2727	2727
query60	303	289	276	276
query61	131	125	119	119
query62	748	740	681	681
query63	230	189	194	189
query64	2841	1072	663	663
query65	4544	4504	4569	4504
query66	778	405	299	299
query67	16388	15606	15494	15494
query68	9702	897	505	505
query69	486	337	264	264
query70	1233	1157	1109	1109
query71	483	292	268	268
query72	5219	4957	4869	4869
query73	702	562	343	343
query74	8975	9128	8738	8738
query75	4478	3270	2743	2743
query76	4445	1195	790	790
query77	991	366	295	295
query78	9915	10094	9364	9364
query79	2433	820	572	572
query80	669	524	444	444
query81	538	267	222	222
query82	477	121	98	98
query83	199	176	161	161
query84	280	92	75	75
query85	742	349	309	309
query86	333	307	280	280
query87	4519	4443	4493	4443
query88	3007	2290	2263	2263
query89	413	313	283	283
query90	1959	215	223	215
query91	137	139	109	109
query92	70	55	55	55
query93	1244	1092	577	577
query94	671	425	305	305
query95	358	274	272	272
query96	487	566	279	279
query97	3346	3445	3348	3348
query98	225	207	205	205
query99	1452	1418	1250	1250
Total cold run time: 278665 ms
Total hot run time: 194532 ms

@doris-robot
Copy link

doris-robot commented Mar 25, 2025

ClickBench: Total hot run time: 31.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2fd4fd2f9e461390767043b6c2a72b07358a54a9, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.10
query3	0.25	0.19	0.19
query4	1.59	0.18	0.20
query5	0.57	0.58	0.59
query6	1.20	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.57	0.53	0.54
query10	0.55	0.59	0.56
query11	0.15	0.11	0.10
query12	0.14	0.12	0.12
query13	0.62	0.61	0.60
query14	2.70	2.80	2.68
query15	0.93	0.84	0.84
query16	0.37	0.38	0.37
query17	1.03	1.02	1.01
query18	0.21	0.19	0.19
query19	1.86	1.97	1.84
query20	0.02	0.01	0.01
query21	15.36	0.93	0.53
query22	0.79	1.25	0.67
query23	14.82	1.40	0.58
query24	6.81	1.10	0.99
query25	0.47	0.16	0.07
query26	0.56	0.16	0.13
query27	0.06	0.05	0.05
query28	10.11	0.94	0.43
query29	12.54	3.98	3.32
query30	0.24	0.09	0.06
query31	2.85	0.59	0.38
query32	3.23	0.54	0.47
query33	3.01	3.05	3.13
query34	15.77	5.16	4.54
query35	4.58	4.54	4.56
query36	0.68	0.50	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.36 s
Total hot run time: 31.32 s

@englefly
Copy link
Contributor Author

englefly commented Mar 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 25, 2025

TPC-H: Total hot run time: 35113 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3d3b57114c849b62ba6a716d30a84147db7337de, data reload: false

------ Round 1 ----------------------------------
q1	26263	5121	5106	5106
q2	2063	297	168	168
q3	10549	1284	700	700
q4	10293	1054	577	577
q5	8401	2439	2343	2343
q6	197	168	136	136
q7	1037	751	614	614
q8	9325	1346	1091	1091
q9	6996	5112	5046	5046
q10	6816	2318	1946	1946
q11	495	276	262	262
q12	357	358	223	223
q13	17761	3648	3102	3102
q14	226	227	209	209
q15	551	487	500	487
q16	617	610	592	592
q17	582	871	342	342
q18	7654	7237	7191	7191
q19	1411	974	588	588
q20	331	325	188	188
q21	4436	3442	3255	3255
q22	1003	1032	947	947
Total cold run time: 117364 ms
Total hot run time: 35113 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5230	5375	5161	5161
q2	232	328	229	229
q3	2136	2678	2277	2277
q4	1481	1957	1494	1494
q5	4602	4418	4375	4375
q6	212	165	124	124
q7	1927	1925	1741	1741
q8	2595	2593	2598	2593
q9	7168	7140	7189	7140
q10	2975	3171	2771	2771
q11	576	501	491	491
q12	660	742	654	654
q13	3552	3915	3219	3219
q14	292	294	264	264
q15	562	486	493	486
q16	667	675	643	643
q17	1184	1458	1432	1432
q18	7679	7469	7380	7380
q19	863	906	919	906
q20	2011	2013	1874	1874
q21	5230	4884	4815	4815
q22	1094	1055	953	953
Total cold run time: 52928 ms
Total hot run time: 51022 ms

@doris-robot
Copy link

doris-robot commented Mar 25, 2025

TPC-DS: Total hot run time: 186994 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3d3b57114c849b62ba6a716d30a84147db7337de, data reload: false

query1	1016	501	476	476
query2	6543	1999	1971	1971
query3	6795	228	220	220
query4	26191	23753	23063	23063
query5	4375	666	478	478
query6	292	201	191	191
query7	4609	489	282	282
query8	295	267	241	241
query9	8627	2618	2616	2616
query10	481	328	268	268
query11	15701	15324	14839	14839
query12	170	114	108	108
query13	1672	524	399	399
query14	9546	6234	6286	6234
query15	205	190	172	172
query16	7299	621	447	447
query17	1196	704	562	562
query18	1979	416	322	322
query19	190	190	160	160
query20	125	122	119	119
query21	212	123	104	104
query22	4418	4442	4303	4303
query23	33962	32920	33093	32920
query24	8262	2415	2381	2381
query25	530	468	389	389
query26	1229	265	141	141
query27	2725	486	318	318
query28	4390	2402	2401	2401
query29	764	591	424	424
query30	284	222	194	194
query31	963	854	760	760
query32	73	66	66	66
query33	560	375	311	311
query34	781	863	539	539
query35	830	823	739	739
query36	959	998	881	881
query37	118	104	77	77
query38	4232	4081	4239	4081
query39	1462	1416	1383	1383
query40	203	121	103	103
query41	58	55	54	54
query42	122	106	106	106
query43	512	501	467	467
query44	1347	817	795	795
query45	175	176	175	175
query46	831	1020	614	614
query47	1732	1808	1710	1710
query48	381	403	298	298
query49	776	502	435	435
query50	707	736	418	418
query51	4214	4144	4175	4144
query52	116	110	106	106
query53	225	260	179	179
query54	485	487	411	411
query55	82	81	83	81
query56	280	302	265	265
query57	1153	1146	1087	1087
query58	250	268	240	240
query59	2669	2799	2688	2688
query60	290	283	257	257
query61	125	125	129	125
query62	807	728	665	665
query63	221	185	179	179
query64	4371	1008	674	674
query65	4457	4355	4392	4355
query66	1136	408	293	293
query67	15809	15812	15470	15470
query68	8201	904	496	496
query69	459	328	278	278
query70	1223	1103	1117	1103
query71	454	302	271	271
query72	5598	5149	5266	5149
query73	729	687	347	347
query74	9058	9133	8613	8613
query75	3834	3247	2760	2760
query76	3696	1200	776	776
query77	772	396	285	285
query78	10046	10294	9350	9350
query79	1944	805	604	604
query80	639	522	441	441
query81	460	258	226	226
query82	448	126	98	98
query83	220	175	153	153
query84	283	99	77	77
query85	764	351	305	305
query86	348	309	276	276
query87	4515	4468	4342	4342
query88	3355	2190	2243	2190
query89	389	313	290	290
query90	1947	209	214	209
query91	146	138	110	110
query92	76	61	58	58
query93	1449	1080	581	581
query94	673	424	326	326
query95	357	277	271	271
query96	481	573	280	280
query97	3370	3419	3296	3296
query98	225	206	198	198
query99	1459	1400	1261	1261
Total cold run time: 275213 ms
Total hot run time: 186994 ms

@doris-robot
Copy link

doris-robot commented Mar 25, 2025

ClickBench: Total hot run time: 30.94 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3d3b57114c849b62ba6a716d30a84147db7337de, data reload: false

query1	0.03	0.04	0.03
query2	0.13	0.10	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.19
query5	0.58	0.56	0.58
query6	1.19	0.70	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.53	0.51
query10	0.58	0.58	0.56
query11	0.17	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.61	0.60
query14	2.69	2.68	2.73
query15	0.92	0.85	0.85
query16	0.38	0.38	0.37
query17	1.02	1.07	1.04
query18	0.22	0.19	0.19
query19	1.89	1.92	1.90
query20	0.02	0.01	0.02
query21	15.36	0.88	0.55
query22	0.77	1.21	0.72
query23	14.81	1.38	0.65
query24	7.05	1.75	0.45
query25	0.45	0.20	0.14
query26	0.65	0.16	0.13
query27	0.05	0.05	0.05
query28	9.12	0.91	0.42
query29	12.54	4.03	3.33
query30	0.24	0.09	0.07
query31	2.82	0.59	0.40
query32	3.22	0.55	0.48
query33	3.11	3.00	3.04
query34	15.84	5.08	4.51
query35	4.50	4.55	4.50
query36	0.67	0.49	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.74 s
Total hot run time: 30.94 s

@englefly englefly force-pushed the no-exception-stats-calc branch from 3d3b571 to 7547fe3 Compare March 26, 2025 01:38
@englefly
Copy link
Contributor Author

englefly commented Mar 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-H: Total hot run time: 34212 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7547fe3319e599cc18372d5e9ac4a27eae7f04b1, data reload: false

------ Round 1 ----------------------------------
q1	26396	5062	5022	5022
q2	2073	294	166	166
q3	10409	1202	682	682
q4	10242	1002	535	535
q5	7526	2332	2326	2326
q6	189	160	134	134
q7	913	710	604	604
q8	9321	1277	1123	1123
q9	6840	5144	5138	5138
q10	6816	2301	1892	1892
q11	479	279	268	268
q12	363	359	221	221
q13	17781	3692	3065	3065
q14	231	232	216	216
q15	539	491	493	491
q16	618	633	601	601
q17	583	871	348	348
q18	7493	7104	7198	7104
q19	1519	953	597	597
q20	328	325	196	196
q21	4486	3447	2491	2491
q22	1044	1002	992	992
Total cold run time: 116189 ms
Total hot run time: 34212 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5193	5101	5141	5101
q2	241	328	236	236
q3	2132	2656	2290	2290
q4	1446	1791	1520	1520
q5	4407	4368	4390	4368
q6	219	166	128	128
q7	1974	1926	1772	1772
q8	2564	2637	2549	2549
q9	7324	7224	7133	7133
q10	2975	3170	2764	2764
q11	581	510	524	510
q12	690	778	660	660
q13	3582	4059	3460	3460
q14	305	335	266	266
q15	532	479	479	479
q16	672	718	689	689
q17	1211	1542	1424	1424
q18	7914	7966	7639	7639
q19	890	859	877	859
q20	1967	2026	1948	1948
q21	5530	4839	4706	4706
q22	1085	1034	1021	1021
Total cold run time: 53434 ms
Total hot run time: 51522 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-DS: Total hot run time: 193268 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7547fe3319e599cc18372d5e9ac4a27eae7f04b1, data reload: false

query1	1398	1088	1061	1061
query2	6416	1931	1938	1931
query3	11011	4599	4518	4518
query4	26148	23772	23241	23241
query5	3859	662	488	488
query6	306	201	194	194
query7	3984	490	292	292
query8	300	242	231	231
query9	8463	2550	2539	2539
query10	465	302	247	247
query11	15425	15274	14890	14890
query12	162	111	105	105
query13	1558	533	390	390
query14	8546	6185	6384	6185
query15	209	199	174	174
query16	7170	653	489	489
query17	1185	699	575	575
query18	1993	427	323	323
query19	202	208	161	161
query20	123	124	118	118
query21	218	123	105	105
query22	4612	4729	4636	4636
query23	34473	33386	33592	33386
query24	8917	2437	2402	2402
query25	522	482	404	404
query26	1255	268	142	142
query27	3003	503	320	320
query28	4653	2468	2432	2432
query29	789	565	425	425
query30	278	229	214	214
query31	928	881	810	810
query32	71	69	69	69
query33	547	369	320	320
query34	800	891	508	508
query35	839	833	758	758
query36	977	1007	897	897
query37	127	102	76	76
query38	4228	4349	4171	4171
query39	1509	1429	1416	1416
query40	214	123	102	102
query41	58	56	52	52
query42	119	110	109	109
query43	525	518	492	492
query44	1341	822	817	817
query45	179	179	167	167
query46	849	1046	635	635
query47	1853	1887	1800	1800
query48	397	440	322	322
query49	786	564	459	459
query50	707	750	426	426
query51	4329	4348	4242	4242
query52	117	108	94	94
query53	233	255	180	180
query54	529	497	427	427
query55	90	77	84	77
query56	274	285	267	267
query57	1222	1212	1136	1136
query58	266	252	250	250
query59	2802	2802	2691	2691
query60	309	282	268	268
query61	136	125	131	125
query62	790	731	698	698
query63	218	181	189	181
query64	4159	1048	707	707
query65	4613	4372	4382	4372
query66	1101	406	324	324
query67	16321	15617	15571	15571
query68	7574	808	488	488
query69	443	304	262	262
query70	1160	1132	1094	1094
query71	380	278	255	255
query72	5854	4783	4718	4718
query73	545	618	346	346
query74	9294	9067	8950	8950
query75	3305	3373	2733	2733
query76	2794	1192	792	792
query77	739	359	279	279
query78	10312	10238	9270	9270
query79	1931	806	552	552
query80	663	513	444	444
query81	503	259	225	225
query82	218	125	96	96
query83	175	235	158	158
query84	244	103	73	73
query85	761	360	315	315
query86	365	283	281	281
query87	4444	4504	4437	4437
query88	2999	2206	2206	2206
query89	380	309	278	278
query90	1934	207	211	207
query91	139	144	111	111
query92	77	60	60	60
query93	2154	1074	571	571
query94	635	386	270	270
query95	347	277	261	261
query96	479	559	272	272
query97	3109	3247	3153	3153
query98	235	207	199	199
query99	1327	1389	1278	1278
Total cold run time: 277725 ms
Total hot run time: 193268 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

ClickBench: Total hot run time: 31.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7547fe3319e599cc18372d5e9ac4a27eae7f04b1, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.25	0.20	0.20
query4	1.59	0.19	0.19
query5	0.60	0.59	0.59
query6	1.19	0.72	0.70
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.58	0.52	0.54
query10	0.57	0.61	0.57
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.60	0.60
query14	2.82	2.70	2.79
query15	0.96	0.86	0.86
query16	0.38	0.38	0.39
query17	1.05	1.04	1.05
query18	0.22	0.21	0.20
query19	1.94	1.99	1.81
query20	0.01	0.01	0.01
query21	15.38	0.93	0.53
query22	0.74	1.21	0.73
query23	14.85	1.40	0.67
query24	7.68	1.47	0.57
query25	0.49	0.21	0.16
query26	0.62	0.17	0.13
query27	0.06	0.05	0.05
query28	9.09	0.91	0.45
query29	12.86	3.95	3.33
query30	0.24	0.08	0.05
query31	2.84	0.59	0.38
query32	3.23	0.57	0.47
query33	3.06	3.02	3.16
query34	15.66	5.12	4.49
query35	4.54	4.53	4.55
query36	0.70	0.50	0.50
query37	0.08	0.06	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.83 s
Total hot run time: 31.14 s

morrySnow
morrySnow reviewed Mar 26, 2025
View reviewed changes
@englefly
Copy link
Contributor Author

englefly commented Mar 27, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-H: Total hot run time: 34147 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7bdce7b1863fd85a774dfa2784ade0cee3b84eeb, data reload: false

------ Round 1 ----------------------------------
q1	25771	5025	5038	5025
q2	2076	293	163	163
q3	10412	1272	677	677
q4	10219	1025	545	545
q5	7796	2380	2346	2346
q6	186	164	132	132
q7	937	750	618	618
q8	9316	1225	1104	1104
q9	6916	5146	5064	5064
q10	6815	2316	1892	1892
q11	466	269	262	262
q12	351	351	217	217
q13	17805	3943	3125	3125
q14	229	229	209	209
q15	530	486	485	485
q16	629	632	593	593
q17	578	856	347	347
q18	7810	7318	7147	7147
q19	1486	945	564	564
q20	316	336	195	195
q21	3823	2600	2456	2456
q22	1050	1047	981	981
Total cold run time: 115517 ms
Total hot run time: 34147 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5152	5097	5131	5097
q2	236	328	232	232
q3	2145	2676	2314	2314
q4	1455	1832	1406	1406
q5	4570	4518	4411	4411
q6	221	163	126	126
q7	1987	1897	1731	1731
q8	2613	2477	2508	2477
q9	7223	7202	7203	7202
q10	2941	3225	2735	2735
q11	557	497	477	477
q12	711	768	639	639
q13	3486	3887	3244	3244
q14	275	303	268	268
q15	531	465	485	465
q16	636	664	649	649
q17	1144	1529	1339	1339
q18	7855	7406	7418	7406
q19	808	820	904	820
q20	1979	2008	1859	1859
q21	5231	4722	4617	4617
q22	1103	1046	985	985
Total cold run time: 52859 ms
Total hot run time: 50499 ms

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-DS: Total hot run time: 185909 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7bdce7b1863fd85a774dfa2784ade0cee3b84eeb, data reload: false

query1	1017	483	467	467
query2	6548	1887	1900	1887
query3	6789	220	217	217
query4	26970	23862	23187	23187
query5	4345	646	470	470
query6	316	204	189	189
query7	4612	474	275	275
query8	301	261	237	237
query9	8614	2540	2517	2517
query10	477	316	251	251
query11	15576	15195	14776	14776
query12	155	109	101	101
query13	1649	507	391	391
query14	8637	6131	6122	6122
query15	191	187	165	165
query16	7125	620	433	433
query17	910	703	552	552
query18	1961	386	294	294
query19	187	174	152	152
query20	123	113	113	113
query21	208	122	105	105
query22	4217	4271	4202	4202
query23	33830	33195	33065	33065
query24	8464	2351	2323	2323
query25	528	446	399	399
query26	1227	260	147	147
query27	2773	499	317	317
query28	4430	2430	2388	2388
query29	740	563	417	417
query30	282	220	187	187
query31	929	842	804	804
query32	74	67	63	63
query33	546	365	313	313
query34	777	848	502	502
query35	795	830	775	775
query36	955	988	882	882
query37	126	99	81	81
query38	4139	4253	4080	4080
query39	1460	1387	1437	1387
query40	208	123	114	114
query41	61	60	59	59
query42	121	105	107	105
query43	492	516	463	463
query44	1289	802	800	800
query45	178	176	173	173
query46	827	1027	631	631
query47	1753	1793	1735	1735
query48	378	420	308	308
query49	810	541	464	464
query50	670	741	408	408
query51	4154	4210	4159	4159
query52	113	113	144	113
query53	227	255	178	178
query54	488	483	409	409
query55	88	83	83	83
query56	277	275	262	262
query57	1133	1143	1097	1097
query58	239	243	237	237
query59	2748	2876	2545	2545
query60	282	276	263	263
query61	129	122	125	122
query62	792	718	645	645
query63	212	180	179	179
query64	4291	991	666	666
query65	4416	4307	4284	4284
query66	1142	404	301	301
query67	15871	15739	15498	15498
query68	9018	888	522	522
query69	471	296	258	258
query70	1208	1146	1075	1075
query71	453	288	267	267
query72	5469	4627	4872	4627
query73	714	594	353	353
query74	8875	9121	8700	8700
query75	4243	3222	2743	2743
query76	3644	1183	742	742
query77	938	371	287	287
query78	10138	10202	9398	9398
query79	1803	803	567	567
query80	615	524	450	450
query81	464	259	230	230
query82	453	128	100	100
query83	181	175	157	157
query84	289	103	76	76
query85	839	367	321	321
query86	346	310	310	310
query87	4505	4602	4235	4235
query88	3402	2231	2279	2231
query89	393	314	295	295
query90	1928	206	214	206
query91	144	141	108	108
query92	81	58	58	58
query93	1492	1044	579	579
query94	683	426	299	299
query95	357	271	261	261
query96	477	562	274	274
query97	3193	3233	3166	3166
query98	234	209	198	198
query99	1428	1393	1282	1282
Total cold run time: 274870 ms
Total hot run time: 185909 ms

@englefly
Copy link
Contributor Author

englefly commented Apr 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 2, 2025

TPC-H: Total hot run time: 34058 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit db7e505667fd0a02672b7000885cb57365a0b4cc, data reload: false

------ Round 1 ----------------------------------
q1	25630	5057	5064	5057
q2	2094	303	183	183
q3	10379	1240	671	671
q4	10374	990	517	517
q5	7653	2335	2298	2298
q6	187	163	130	130
q7	902	759	591	591
q8	9314	1279	1146	1146
q9	6815	5135	5066	5066
q10	6815	2276	1887	1887
q11	466	292	268	268
q12	344	349	221	221
q13	17766	3679	3051	3051
q14	220	226	213	213
q15	519	503	488	488
q16	619	628	594	594
q17	597	842	380	380
q18	7551	7169	7056	7056
q19	1212	956	554	554
q20	336	331	227	227
q21	4149	2634	2487	2487
q22	1032	1024	973	973
Total cold run time: 114974 ms
Total hot run time: 34058 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5156	5127	5097	5097
q2	235	326	229	229
q3	2168	2655	2256	2256
q4	1390	1824	1388	1388
q5	4409	4377	4375	4375
q6	214	170	127	127
q7	2006	1927	1782	1782
q8	2560	2526	2517	2517
q9	7250	7176	6971	6971
q10	3002	3184	2737	2737
q11	590	512	487	487
q12	710	783	606	606
q13	3538	3898	3280	3280
q14	284	295	298	295
q15	520	474	486	474
q16	671	671	641	641
q17	1156	1526	1392	1392
q18	7938	7642	7450	7450
q19	829	832	1042	832
q20	1934	2027	1826	1826
q21	5185	4860	4842	4842
q22	1077	1038	981	981
Total cold run time: 52822 ms
Total hot run time: 50585 ms

@doris-robot
Copy link

doris-robot commented Apr 2, 2025

TPC-DS: Total hot run time: 193356 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit db7e505667fd0a02672b7000885cb57365a0b4cc, data reload: false

query1	1420	1062	1040	1040
query2	6253	1946	1922	1922
query3	11016	4612	4427	4427
query4	53051	24978	23548	23548
query5	4902	622	461	461
query6	331	204	206	204
query7	4886	520	281	281
query8	326	254	249	249
query9	5352	2584	2595	2584
query10	392	340	292	292
query11	15083	14962	14759	14759
query12	158	110	105	105
query13	1024	534	390	390
query14	10189	6473	6608	6473
query15	207	194	178	178
query16	7115	656	511	511
query17	1089	768	623	623
query18	1629	416	356	356
query19	205	204	188	188
query20	140	125	121	121
query21	218	131	117	117
query22	4363	4539	4169	4169
query23	34066	33334	33201	33201
query24	6680	2472	2428	2428
query25	452	471	402	402
query26	749	273	153	153
query27	2638	504	336	336
query28	3193	2436	2446	2436
query29	595	582	451	451
query30	273	225	198	198
query31	862	876	793	793
query32	75	66	64	64
query33	469	372	308	308
query34	788	868	534	534
query35	796	865	764	764
query36	961	1009	910	910
query37	120	102	77	77
query38	4080	4210	4145	4145
query39	1532	1398	1431	1398
query40	215	122	108	108
query41	52	53	51	51
query42	126	105	103	103
query43	510	516	475	475
query44	1344	819	830	819
query45	182	187	175	175
query46	878	1028	682	682
query47	1893	1900	1794	1794
query48	385	439	319	319
query49	701	520	441	441
query50	724	704	415	415
query51	4327	4368	4253	4253
query52	109	106	101	101
query53	238	272	191	191
query54	595	591	535	535
query55	84	83	86	83
query56	295	291	289	289
query57	1156	1247	1113	1113
query58	269	288	267	267
query59	2677	2797	2757	2757
query60	335	330	324	324
query61	170	125	124	124
query62	743	747	653	653
query63	233	194	190	190
query64	2220	1047	699	699
query65	4440	4356	4410	4356
query66	789	406	306	306
query67	16039	15557	15359	15359
query68	7074	840	515	515
query69	526	316	267	267
query70	1205	1108	1113	1108
query71	486	330	300	300
query72	5938	4778	4850	4778
query73	1496	652	350	350
query74	8892	8944	8862	8862
query75	4061	3179	2731	2731
query76	4301	1194	758	758
query77	854	385	283	283
query78	9981	10131	9306	9306
query79	2587	817	564	564
query80	602	495	448	448
query81	482	255	237	237
query82	447	124	96	96
query83	290	248	231	231
query84	295	118	88	88
query85	776	354	329	329
query86	373	309	282	282
query87	4494	4581	4324	4324
query88	3504	2230	2217	2217
query89	406	316	278	278
query90	1963	215	211	211
query91	144	152	113	113
query92	71	63	56	56
query93	1720	947	583	583
query94	665	417	311	311
query95	373	303	286	286
query96	486	563	276	276
query97	3184	3196	3118	3118
query98	232	204	202	202
query99	1403	1398	1270	1270
Total cold run time: 298718 ms
Total hot run time: 193356 ms

@doris-robot
Copy link

doris-robot commented Apr 2, 2025

ClickBench: Total hot run time: 31.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit db7e505667fd0a02672b7000885cb57365a0b4cc, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.25	0.20	0.19
query4	1.59	0.20	0.20
query5	0.61	0.56	0.60
query6	1.18	0.70	0.73
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.58	0.51	0.52
query10	0.59	0.60	0.57
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.59	0.59
query14	2.71	2.70	2.82
query15	0.94	0.85	0.85
query16	0.38	0.38	0.40
query17	1.01	0.99	1.01
query18	0.21	0.19	0.19
query19	1.91	1.92	1.88
query20	0.01	0.01	0.01
query21	15.36	0.93	0.56
query22	0.76	1.13	0.65
query23	15.02	1.38	0.66
query24	7.38	0.95	1.17
query25	0.53	0.20	0.07
query26	0.55	0.16	0.13
query27	0.05	0.05	0.05
query28	9.85	0.89	0.43
query29	12.60	3.98	3.24
query30	0.24	0.09	0.06
query31	2.84	0.58	0.38
query32	3.22	0.54	0.46
query33	3.01	3.02	3.03
query34	15.81	5.09	4.51
query35	4.50	4.50	4.50
query36	0.68	0.50	0.47
query37	0.08	0.06	0.06
query38	0.06	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.14
query41	0.09	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.03 s
Total hot run time: 31.17 s

@englefly
Copy link
Contributor Author

englefly commented Apr 3, 2025

run feut

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2025

PR approved by anyone and no changes requested.

yujun777
yujun777 approved these changes Apr 3, 2025
View reviewed changes
Copy link
Contributor

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@englefly englefly merged commit 7d67b39 into apache:master Apr 3, 2025
24 of 26 checks passed
@englefly englefly deleted the no-exception-stats-calc branch April 3, 2025 09:51
englefly added a commit to englefly/incubator-doris that referenced this pull request Apr 24, 2025
@englefly
[opt](nereids) catch all exceptions in StatsCalculator (apache#49415)
6375812 
catch all exceptions in ExpressionEstimation so that any error in
ExpressionEstimation does not break the query.
@englefly englefly mentioned this pull request Apr 24, 2025
Merged
16 tasks
englefly added a commit to englefly/incubator-doris that referenced this pull request Apr 24, 2025
@englefly
[opt](nereids) catch all exceptions in StatsCalculator (apache#49415)
912a103 
catch all exceptions in ExpressionEstimation so that any error in
ExpressionEstimation does not break the query.
@englefly englefly mentioned this pull request Apr 24, 2025
Merged
16 tasks
yiguolei pushed a commit that referenced this pull request May 9, 2025
@englefly
branch-2.1 [opt](nereids) catch all exceptions in StatsCalculator (#4…
fde8d05 
…9415) (#50364)
@yiguolei yiguolei mentioned this pull request May 13, 2025
Closed
dataroaring pushed a commit that referenced this pull request May 14, 2025
@englefly
branch-3.0 [opt](nereids) catch all exceptions in StatsCalculator (#4…
669f2df 
…9415) (#50360)

catch all exceptions in ExpressionEstimation so that any error in
ExpressionEstimation does not break the query.

### What problem does this PR solve?
pick #49415
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@englefly @koarz
[opt](nereids) catch all exceptions in StatsCalculator (apache#49415)
ce57711 
### What problem does this PR solve?
catch all exceptions in ExpressionEstimation so that any error in
ExpressionEstimation does not break the query.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

+1 more reviewer

@yujun777 yujun777 yujun777 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.6-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@englefly @hello-stephen @doris-robot @yujun777 @morrySnow @yiguolei @dataroaring