Skip to content

[chore](log) Reduce some BE execution log #42383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
BiteTheDDDDt merged 1 commit into from
Oct 28, 2024
Merged

[chore](log) Reduce some BE execution log #42383

BiteTheDDDDt merged 1 commit into from
Oct 28, 2024

Conversation

@zclllyybb
Copy link
Contributor

@zclllyybb zclllyybb commented Oct 24, 2024

Proposed changes

Issue Number: close #xxx

  1. timeout does not print stack
  2. when the runtime filter's relative query is finished before, it does not print the log.

@doris-robot
Copy link

doris-robot commented Oct 24, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb
Copy link
Contributor Author

zclllyybb commented Oct 24, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Oct 24, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Oct 24, 2024

TPC-H: Total hot run time: 41537 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ca52ac14a72c39671954d9c1eeea88ced0663c3, data reload: false

------ Round 1 ----------------------------------
q1	17734	7624	7344	7344
q2	2057	171	151	151
q3	10656	1172	1263	1172
q4	10441	807	875	807
q5	7775	3305	3095	3095
q6	242	155	155	155
q7	1027	628	628	628
q8	9371	1982	2001	1982
q9	6610	6467	6439	6439
q10	7041	2466	2473	2466
q11	451	261	267	261
q12	408	221	224	221
q13	17807	2982	3010	2982
q14	249	209	215	209
q15	574	529	519	519
q16	645	573	606	573
q17	963	490	483	483
q18	7568	6764	6795	6764
q19	1367	931	980	931
q20	457	189	183	183
q21	4005	3154	3161	3154
q22	1114	1018	1020	1018
Total cold run time: 108562 ms
Total hot run time: 41537 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7275	7324	7307	7307
q2	319	225	224	224
q3	2932	2828	2818	2818
q4	1977	1734	1694	1694
q5	5442	5473	5563	5473
q6	231	143	143	143
q7	2174	1743	1699	1699
q8	3239	3408	3467	3408
q9	8620	8560	8590	8560
q10	3531	3485	3451	3451
q11	577	480	481	480
q12	795	608	607	607
q13	8268	3026	3001	3001
q14	300	267	261	261
q15	575	519	520	519
q16	683	634	652	634
q17	1846	1589	1588	1588
q18	7856	7507	7298	7298
q19	1684	1390	1591	1390
q20	2083	1812	1831	1812
q21	5509	5423	5370	5370
q22	1120	1048	1014	1014
Total cold run time: 67036 ms
Total hot run time: 58751 ms

@doris-robot
Copy link

doris-robot commented Oct 24, 2024

TeamCity be ut coverage result:
Function Coverage: 37.48% (9711/25913)
Line Coverage: 28.74% (80530/280246)
Region Coverage: 28.17% (41642/147837)
Branch Coverage: 24.73% (21153/85538)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3ca52ac14a72c39671954d9c1eeea88ced0663c3_3ca52ac14a72c39671954d9c1eeea88ced0663c3/report/index.html

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 25, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Oct 25, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 25, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Oct 25, 2024

TPC-H: Total hot run time: 41365 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ca52ac14a72c39671954d9c1eeea88ced0663c3, data reload: false

------ Round 1 ----------------------------------
q1	17792	7415	7277	7277
q2	2048	292	275	275
q3	12295	1068	1210	1068
q4	10564	830	906	830
q5	7754	3150	3074	3074
q6	240	147	142	142
q7	1021	626	603	603
q8	9361	2006	1968	1968
q9	6577	6461	6446	6446
q10	7028	2403	2445	2403
q11	443	247	250	247
q12	408	218	214	214
q13	17772	2992	3040	2992
q14	249	218	214	214
q15	572	511	513	511
q16	660	581	603	581
q17	978	552	517	517
q18	7474	6770	6811	6770
q19	1335	1041	946	946
q20	491	183	182	182
q21	4059	3132	3112	3112
q22	1124	1008	993	993
Total cold run time: 110245 ms
Total hot run time: 41365 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7256	7194	7292	7194
q2	326	236	238	236
q3	3072	2986	2947	2947
q4	2190	1882	1811	1811
q5	5734	5797	5768	5768
q6	221	142	140	140
q7	2251	1838	1806	1806
q8	3385	3574	3455	3455
q9	8942	8922	8882	8882
q10	3583	3536	3588	3536
q11	576	485	503	485
q12	838	615	621	615
q13	9232	3218	3163	3163
q14	314	270	295	270
q15	563	532	525	525
q16	697	644	663	644
q17	1835	1650	1644	1644
q18	8270	7982	7519	7519
q19	1727	1486	1689	1486
q20	2145	1837	1882	1837
q21	5680	5339	5571	5339
q22	1139	1079	1027	1027
Total cold run time: 69976 ms
Total hot run time: 60329 ms

@doris-robot
Copy link

doris-robot commented Oct 25, 2024

TPC-DS: Total hot run time: 192092 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ca52ac14a72c39671954d9c1eeea88ced0663c3, data reload: false

query1	863	388	409	388
query2	6252	2132	2093	2093
query3	8678	190	197	190
query4	35488	23626	23495	23495
query5	4757	437	437	437
query6	269	165	162	162
query7	4195	294	295	294
query8	286	233	243	233
query9	9166	2684	2685	2684
query10	498	258	239	239
query11	18167	15220	15144	15144
query12	155	104	107	104
query13	1572	426	402	402
query14	10085	7134	6940	6940
query15	240	172	172	172
query16	8038	484	494	484
query17	1393	591	585	585
query18	2194	315	299	299
query19	242	158	164	158
query20	124	113	114	113
query21	209	109	109	109
query22	4774	4732	4562	4562
query23	35022	35488	34278	34278
query24	11007	2811	2845	2811
query25	544	397	403	397
query26	726	160	165	160
query27	2277	289	286	286
query28	7387	2421	2443	2421
query29	667	435	411	411
query30	282	165	158	158
query31	1049	790	803	790
query32	88	56	59	56
query33	744	279	287	279
query34	908	522	504	504
query35	1013	900	882	882
query36	1078	955	958	955
query37	131	76	71	71
query38	4358	4291	4219	4219
query39	1480	1410	1424	1410
query40	201	100	100	100
query41	49	47	48	47
query42	116	101	103	101
query43	539	520	489	489
query44	1220	818	812	812
query45	184	166	166	166
query46	1139	678	689	678
query47	1911	1835	1840	1835
query48	437	325	332	325
query49	910	425	410	410
query50	798	382	390	382
query51	7108	7039	6985	6985
query52	101	88	96	88
query53	255	184	181	181
query54	1084	411	425	411
query55	77	75	80	75
query56	265	251	265	251
query57	1284	1182	1138	1138
query58	224	198	199	198
query59	3244	3176	3053	3053
query60	262	240	252	240
query61	102	99	103	99
query62	812	674	673	673
query63	214	191	183	183
query64	3865	636	639	636
query65	3270	3203	3191	3191
query66	730	302	302	302
query67	16080	15675	15808	15675
query68	4492	571	561	561
query69	429	250	257	250
query70	1208	1186	1172	1172
query71	314	268	268	268
query72	6043	4006	3931	3931
query73	761	352	359	352
query74	10015	8931	9007	8931
query75	3376	2672	2690	2672
query76	2789	989	981	981
query77	371	271	263	263
query78	10621	9595	9495	9495
query79	1384	586	598	586
query80	872	430	428	428
query81	546	240	242	240
query82	700	116	116	116
query83	199	145	134	134
query84	243	70	73	70
query85	1231	306	287	287
query86	361	300	304	300
query87	4836	4658	4763	4658
query88	3284	2233	2190	2190
query89	410	284	286	284
query90	1902	185	188	185
query91	137	102	98	98
query92	55	47	51	47
query93	1056	547	540	540
query94	720	282	291	282
query95	345	244	248	244
query96	614	286	284	284
query97	2870	2709	2654	2654
query98	217	207	203	203
query99	1774	1319	1287	1287
Total cold run time: 297062 ms
Total hot run time: 192092 ms

@doris-robot
Copy link

doris-robot commented Oct 25, 2024

ClickBench: Total hot run time: 31.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3ca52ac14a72c39671954d9c1eeea88ced0663c3, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.43	0.41	0.39
query6	1.15	0.65	0.65
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.56	0.51	0.54
query10	0.54	0.54	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.10
query13	0.61	0.59	0.60
query14	2.68	2.74	2.85
query15	0.90	0.84	0.84
query16	0.38	0.37	0.36
query17	1.01	1.00	1.03
query18	0.21	0.21	0.21
query19	1.85	1.87	1.92
query20	0.02	0.01	0.01
query21	15.36	0.58	0.59
query22	2.61	2.56	1.69
query23	17.09	0.91	0.71
query24	3.05	0.45	1.27
query25	0.19	0.17	0.16
query26	0.32	0.14	0.14
query27	0.04	0.04	0.04
query28	11.12	1.10	1.07
query29	12.56	3.33	3.21
query30	0.25	0.06	0.06
query31	2.86	0.39	0.37
query32	3.28	0.46	0.46
query33	3.03	3.05	3.02
query34	17.02	4.50	4.39
query35	4.54	4.49	4.54
query36	0.67	0.48	0.47
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.03
query40	0.15	0.14	0.12
query41	0.07	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.08 s
Total hot run time: 31.8 s

@BiteTheDDDDt BiteTheDDDDt merged commit 76958ed into apache:master Oct 28, 2024
@zclllyybb zclllyybb deleted the reduce_log branch November 8, 2024 03:41
@zclllyybb zclllyybb mentioned this pull request Nov 11, 2024
Merged
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Nov 11, 2024
@zclllyybb
[chore](log) Reduce some BE execution log (apache#42383)
fba4c05 
1. timeout does not print stack
2. when the runtime filter's relative query is finished before, it does
not print the log.
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Nov 11, 2024
@zclllyybb
[chore](log) Reduce some BE execution log (apache#42383)
4eda048 
1. timeout does not print stack
2. when the runtime filter's relative query is finished before, it does
not print the log.
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Nov 11, 2024
@zclllyybb
[chore](log) Reduce some BE execution log (apache#42383)
d29e998 
1. timeout does not print stack
2. when the runtime filter's relative query is finished before, it does
not print the log.
@zclllyybb zclllyybb mentioned this pull request Nov 11, 2024
Merged
yiguolei pushed a commit that referenced this pull request Nov 12, 2024
@zclllyybb
[branch-2.1](log) Pick all BE execution log reduction (#43267)
24b79fd 
pick #42383
pick #42900
yiguolei pushed a commit that referenced this pull request Nov 13, 2024
@zclllyybb
[branch-3.0](log) Pick all BE execution log reduction (#43579)
11a9a48 
pick #42383
pick #42900
@echo-hhj echo-hhj mentioned this pull request Jan 3, 2025
Closed
3 tasks
@wm1581066 wm1581066 added the usercase Important user case type label label Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@Gabriel39 Gabriel39 Gabriel39 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.3-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zclllyybb @doris-robot @BiteTheDDDDt @Gabriel39 @yiguolei @wm1581066