[chore](profile) add task state in profile #42895

mrhhsg · 2024-10-30T03:50:02Z

Proposed changes

Issue Number: close #xxx

doris-robot · 2024-10-30T03:50:06Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

mrhhsg · 2024-10-30T03:50:09Z

run buildall

github-actions · 2024-10-30T03:54:31Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-10-30T05:01:25Z

TeamCity be ut coverage result:
Function Coverage: 37.94% (9845/25950)
Line Coverage: 29.21% (81978/280644)
Region Coverage: 28.55% (42274/148052)
Branch Coverage: 25.08% (21486/85662)
Coverage Report: http://coverage.selectdb-in.cc/coverage/393728005d4634eded6108496d0678ec9da7c748_393728005d4634eded6108496d0678ec9da7c748/report/index.html

doris-robot · 2024-10-30T05:25:10Z

TPC-H: Total hot run time: 41397 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 393728005d4634eded6108496d0678ec9da7c748, data reload: false

------ Round 1 ----------------------------------
q1	17573	7463	7250	7250
q2	2045	158	153	153
q3	10642	1106	1196	1106
q4	10245	867	908	867
q5	7756	3072	2999	2999
q6	236	145	142	142
q7	981	604	597	597
q8	9338	1982	2078	1982
q9	6631	6460	6568	6460
q10	7111	2429	2438	2429
q11	455	253	245	245
q12	408	213	214	213
q13	17785	3007	3009	3007
q14	263	223	208	208
q15	555	511	514	511
q16	664	578	582	578
q17	976	533	631	533
q18	7370	6791	6857	6791
q19	1317	1054	980	980
q20	481	178	179	178
q21	4013	3248	3153	3153
q22	1121	1015	1021	1015
Total cold run time: 107966 ms
Total hot run time: 41397 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7246	7235	7226	7226
q2	324	229	226	226
q3	3032	2968	2981	2968
q4	2030	1817	1773	1773
q5	5738	5783	5792	5783
q6	222	140	139	139
q7	2263	1797	1843	1797
q8	3370	3521	3458	3458
q9	8907	8904	8876	8876
q10	3585	3581	3548	3548
q11	598	489	496	489
q12	842	671	641	641
q13	10922	3185	3169	3169
q14	318	277	270	270
q15	584	535	535	535
q16	688	648	623	623
q17	1844	1630	1614	1614
q18	8447	7774	7566	7566
q19	1722	1420	1496	1420
q20	2123	1880	1868	1868
q21	5579	5322	5316	5316
q22	1150	1026	1071	1026
Total cold run time: 71534 ms
Total hot run time: 60331 ms

doris-robot · 2024-10-30T05:37:20Z

TPC-DS: Total hot run time: 196798 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 393728005d4634eded6108496d0678ec9da7c748, data reload: false

query1	1262	1005	1022	1005
query2	6232	2114	2007	2007
query3	11485	4811	4884	4811
query4	33318	23554	23586	23554
query5	4766	454	426	426
query6	276	170	166	166
query7	3986	285	290	285
query8	281	220	219	219
query9	9410	2706	2698	2698
query10	481	253	260	253
query11	17928	15173	15175	15173
query12	150	104	96	96
query13	1565	429	410	410
query14	9474	7430	7370	7370
query15	263	182	182	182
query16	8037	492	506	492
query17	1476	587	576	576
query18	2188	307	296	296
query19	313	157	156	156
query20	122	125	122	122
query21	205	106	108	106
query22	4643	4564	4466	4466
query23	34671	33772	34003	33772
query24	10752	2773	2795	2773
query25	591	414	404	404
query26	716	157	157	157
query27	2039	280	281	280
query28	6320	2475	2437	2437
query29	752	415	422	415
query30	263	160	163	160
query31	1028	797	835	797
query32	89	58	55	55
query33	682	274	272	272
query34	906	513	507	507
query35	1035	882	874	874
query36	1085	958	952	952
query37	120	82	75	75
query38	4496	4247	4237	4237
query39	1462	1407	1434	1407
query40	202	98	97	97
query41	47	45	48	45
query42	109	96	94	94
query43	534	484	473	473
query44	1136	800	795	795
query45	183	161	166	161
query46	1109	672	699	672
query47	1961	1834	1890	1834
query48	421	322	329	322
query49	909	404	392	392
query50	811	377	391	377
query51	7158	7023	7060	7023
query52	102	92	90	90
query53	249	179	179	179
query54	1146	409	434	409
query55	78	78	77	77
query56	254	246	264	246
query57	1291	1166	1128	1128
query58	213	198	207	198
query59	3159	2984	2885	2885
query60	262	241	244	241
query61	104	100	101	100
query62	851	667	671	667
query63	214	186	180	180
query64	4394	610	615	610
query65	3281	3168	3215	3168
query66	812	294	294	294
query67	15892	15768	15953	15768
query68	4668	565	548	548
query69	436	251	252	251
query70	1182	1109	1083	1083
query71	316	258	250	250
query72	6207	3910	4172	3910
query73	752	350	357	350
query74	10167	9035	8970	8970
query75	3334	2656	2676	2656
query76	2430	915	999	915
query77	367	271	299	271
query78	10510	9628	9502	9502
query79	1596	593	596	593
query80	1142	417	408	408
query81	574	252	246	246
query82	974	113	113	113
query83	234	134	140	134
query84	236	67	69	67
query85	1252	314	285	285
query86	368	302	291	291
query87	4768	4679	4712	4679
query88	3186	2200	2170	2170
query89	409	287	285	285
query90	1988	180	182	180
query91	133	100	98	98
query92	56	49	49	49
query93	1934	551	539	539
query94	888	294	294	294
query95	348	239	243	239
query96	611	285	276	276
query97	2873	2748	2681	2681
query98	211	199	199	199
query99	1563	1337	1298	1298
Total cold run time: 297463 ms
Total hot run time: 196798 ms

doris-robot · 2024-10-30T05:42:49Z

ClickBench: Total hot run time: 32.6 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 393728005d4634eded6108496d0678ec9da7c748, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.65	0.10	0.10
query5	0.41	0.40	0.39
query6	1.15	0.65	0.64
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.57	0.51	0.48
query10	0.55	0.54	0.54
query11	0.14	0.10	0.11
query12	0.13	0.11	0.11
query13	0.60	0.59	0.59
query14	2.72	2.73	2.87
query15	0.90	0.82	0.82
query16	0.37	0.39	0.38
query17	1.05	1.05	0.97
query18	0.21	0.20	0.20
query19	1.94	1.87	1.79
query20	0.01	0.01	0.01
query21	15.35	0.57	0.58
query22	2.26	1.84	2.44
query23	16.95	1.21	0.77
query24	2.58	1.97	1.19
query25	0.29	0.16	0.09
query26	0.44	0.14	0.13
query27	0.03	0.04	0.04
query28	10.19	1.09	1.08
query29	12.54	3.30	3.28
query30	0.24	0.06	0.06
query31	2.86	0.38	0.38
query32	3.28	0.45	0.46
query33	2.98	3.06	3.03
query34	16.96	4.43	4.48
query35	4.47	4.48	4.46
query36	0.67	0.49	0.49
query37	0.08	0.06	0.07
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.37 s
Total hot run time: 32.6 s

yiguolei · 2024-10-30T09:40:32Z

be/src/pipeline/pipeline_task.cpp

            *eos = status.is<ErrorCode::END_OF_FILE>() ? true : *eos;
            if (*eos) { // just return, the scheduler will do finish work
                _eos = true;
+                _task_profile->add_info_string("TaskState", "EOS");


rename to Finished

yiguolei · 2024-10-30T09:41:25Z

be/src/pipeline/exec/scan_operator.cpp

    if (*eos) {
        // reach limit, stop the scanners.
        local_state._scanner_ctx->stop_scanners(state);
+        local_state._scanner_profile->add_info_string("EOS", "true");


mrhhsg · 2024-10-30T09:52:27Z

run buildall

github-actions · 2024-10-30T09:56:54Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-10-30T11:35:19Z

TeamCity be ut coverage result:
Function Coverage: 37.89% (9846/25986)
Line Coverage: 29.16% (81969/281141)
Region Coverage: 28.53% (42304/148254)
Branch Coverage: 25.05% (21488/85782)
Coverage Report: http://coverage.selectdb-in.cc/coverage/45cba10b6279287bd8fbecf01b28fef83b7f2911_45cba10b6279287bd8fbecf01b28fef83b7f2911/report/index.html

mrhhsg · 2024-10-31T00:18:35Z

run buildall

github-actions · 2024-10-31T00:23:00Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-10-31T00:56:16Z

TeamCity be ut coverage result:
Function Coverage: 37.89% (9847/25986)
Line Coverage: 29.16% (81994/281192)
Region Coverage: 28.44% (42329/148831)
Branch Coverage: 25.02% (21498/85908)
Coverage Report: http://coverage.selectdb-in.cc/coverage/42ea6681b2eb835fe0dc887c0a3d72ac0a4231ef_42ea6681b2eb835fe0dc887c0a3d72ac0a4231ef/report/index.html

doris-robot · 2024-10-31T04:42:15Z

TPC-H: Total hot run time: 41378 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 42ea6681b2eb835fe0dc887c0a3d72ac0a4231ef, data reload: false

------ Round 1 ----------------------------------
q1	17573	7404	7351	7351
q2	2046	167	160	160
q3	10567	1072	1182	1072
q4	10509	842	884	842
q5	7747	3044	3068	3044
q6	234	146	152	146
q7	1038	598	600	598
q8	9363	1958	2039	1958
q9	6605	6448	6452	6448
q10	7079	2417	2439	2417
q11	467	257	248	248
q12	402	221	214	214
q13	17787	3021	3044	3021
q14	255	209	206	206
q15	569	513	503	503
q16	657	599	583	583
q17	980	552	583	552
q18	7227	6742	6872	6742
q19	1334	956	1026	956
q20	469	179	185	179
q21	3974	3280	3139	3139
q22	1135	999	1020	999
Total cold run time: 108017 ms
Total hot run time: 41378 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7300	7258	7281	7258
q2	319	229	225	225
q3	3015	2964	2966	2964
q4	2103	1957	1907	1907
q5	5758	5807	5801	5801
q6	225	141	142	141
q7	2212	1851	1802	1802
q8	3353	3558	3426	3426
q9	8990	8879	8899	8879
q10	3579	3555	3578	3555
q11	582	489	474	474
q12	821	617	592	592
q13	10001	3281	3210	3210
q14	304	276	277	276
q15	572	518	516	516
q16	691	638	643	638
q17	1844	1649	1608	1608
q18	8297	7943	7746	7746
q19	1720	1541	1518	1518
q20	2166	1857	1872	1857
q21	5758	5406	5436	5406
q22	1163	1076	1036	1036
Total cold run time: 70773 ms
Total hot run time: 60835 ms

doris-robot · 2024-10-31T04:54:30Z

TPC-DS: Total hot run time: 196688 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 42ea6681b2eb835fe0dc887c0a3d72ac0a4231ef, data reload: false

query1	1251	977	975	975
query2	6251	2064	2024	2024
query3	11631	4988	4808	4808
query4	34245	23706	23693	23693
query5	5358	470	434	434
query6	265	172	181	172
query7	3995	293	290	290
query8	277	221	214	214
query9	9511	2627	2639	2627
query10	526	256	247	247
query11	18450	15344	15247	15247
query12	150	103	96	96
query13	1579	411	431	411
query14	9533	7242	6805	6805
query15	265	186	187	186
query16	8026	486	500	486
query17	1542	605	589	589
query18	2125	306	309	306
query19	274	162	169	162
query20	119	117	117	117
query21	208	114	114	114
query22	4939	4568	4323	4323
query23	34870	34309	33965	33965
query24	10998	2805	2813	2805
query25	610	402	407	402
query26	742	176	176	176
query27	2075	283	295	283
query28	6672	2423	2409	2409
query29	775	423	430	423
query30	257	169	161	161
query31	1032	823	823	823
query32	83	55	59	55
query33	776	273	280	273
query34	945	514	530	514
query35	1098	894	880	880
query36	1111	915	972	915
query37	125	76	76	76
query38	4420	4376	4308	4308
query39	1457	1462	1422	1422
query40	199	100	101	100
query41	47	45	46	45
query42	106	99	98	98
query43	529	494	488	488
query44	1251	805	808	805
query45	182	164	171	164
query46	1112	703	684	684
query47	1933	1861	1866	1861
query48	407	333	323	323
query49	904	413	405	405
query50	805	387	401	387
query51	7183	6881	7025	6881
query52	100	88	89	88
query53	249	184	184	184
query54	1107	403	414	403
query55	79	77	78	77
query56	254	240	244	240
query57	1312	1226	1142	1142
query58	213	208	195	195
query59	3282	3008	2880	2880
query60	264	252	243	243
query61	101	100	106	100
query62	832	667	669	667
query63	210	193	192	192
query64	3966	641	617	617
query65	3315	3245	3225	3225
query66	772	334	294	294
query67	16014	15885	15633	15633
query68	4447	560	537	537
query69	425	261	258	258
query70	1193	1125	1135	1125
query71	296	251	244	244
query72	6300	4014	3949	3949
query73	764	355	355	355
query74	10184	8939	9030	8939
query75	3419	2672	2727	2672
query76	2431	1051	1067	1051
query77	369	289	264	264
query78	10785	9548	9526	9526
query79	1588	586	598	586
query80	1130	427	411	411
query81	573	251	247	247
query82	963	116	115	115
query83	231	139	135	135
query84	233	69	70	69
query85	1300	296	299	296
query86	424	301	264	264
query87	4873	4804	4722	4722
query88	3242	2212	2151	2151
query89	423	285	319	285
query90	1998	192	185	185
query91	133	101	99	99
query92	61	47	47	47
query93	1950	544	534	534
query94	913	288	297	288
query95	348	248	247	247
query96	624	280	288	280
query97	2891	2737	2688	2688
query98	217	199	196	196
query99	1564	1286	1308	1286
Total cold run time: 301586 ms
Total hot run time: 196688 ms

doris-robot · 2024-10-31T05:00:01Z

ClickBench: Total hot run time: 33.08 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 42ea6681b2eb835fe0dc887c0a3d72ac0a4231ef, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.42	0.41	0.40
query6	1.15	0.67	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.48	0.49
query10	0.53	0.54	0.55
query11	0.13	0.10	0.10
query12	0.15	0.11	0.13
query13	0.61	0.60	0.60
query14	2.77	2.82	2.85
query15	0.91	0.82	0.82
query16	0.38	0.38	0.38
query17	0.99	1.07	1.03
query18	0.20	0.20	0.19
query19	1.95	1.90	2.00
query20	0.01	0.02	0.01
query21	15.34	0.60	0.57
query22	2.35	2.05	2.81
query23	16.83	1.05	0.90
query24	3.19	1.59	1.05
query25	0.19	0.17	0.04
query26	0.49	0.14	0.14
query27	0.06	0.04	0.04
query28	10.44	1.11	1.09
query29	12.58	3.34	3.29
query30	0.25	0.06	0.05
query31	2.89	0.40	0.40
query32	3.24	0.46	0.46
query33	2.98	2.96	3.07
query34	17.05	4.47	4.54
query35	4.66	4.48	4.56
query36	0.69	0.49	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.52 s
Total hot run time: 33.08 s

github-actions · 2024-10-31T06:09:44Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-31T06:09:45Z

PR approved by anyone and no changes requested.

## Proposed changes Issue Number: close #xxx

Issue Number: close #xxx

## Proposed changes pick #42895

PR Body: ## Proposed changes Issue Number: close #xxx  Cherry-picked from #42895 Co-authored-by: Jerry Hu <mrhhsg@gmail.com>

yiguolei reviewed Oct 30, 2024

View reviewed changes

mrhhsg force-pushed the add_task_state branch from 3937280 to 45cba10 Compare October 30, 2024 09:52

yiguolei added dev/2.1.x dev/3.0.x labels Oct 30, 2024

[chore](profile) add task state in profile

42ea668

mrhhsg force-pushed the add_task_state branch from 45cba10 to 42ea668 Compare October 31, 2024 00:18

yiguolei approved these changes Oct 31, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 31, 2024

github-actions bot added the reviewed label Oct 31, 2024

jacktengg approved these changes Oct 31, 2024

View reviewed changes

yiguolei merged commit b74e38c into apache:master Oct 31, 2024

github-actions bot pushed a commit that referenced this pull request Oct 31, 2024

[chore](profile) add task state in profile (#42895)

1909ea8

## Proposed changes Issue Number: close #xxx

github-actions bot mentioned this pull request Oct 31, 2024

branch-3.0: [chore](profile) add task state in profile #42981

Merged

mrhhsg deleted the add_task_state branch October 31, 2024 09:54

mrhhsg added a commit to mrhhsg/doris that referenced this pull request Oct 31, 2024

[chore](profile) add task state in profile (apache#42895)

4fe2277

Issue Number: close #xxx

mrhhsg mentioned this pull request Oct 31, 2024

[chore](profile) add task state in profile (#42895) #43013

Merged

yiguolei pushed a commit that referenced this pull request Nov 1, 2024

[chore](profile) add task state in profile (#42895) (#43013)

efd9866

## Proposed changes pick #42895

yiguolei added dev/2.1.7-merged and removed dev/2.1.x labels Nov 1, 2024

dataroaring added dev/3.0.3-merged and removed dev/3.0.x labels Nov 7, 2024

[chore](profile) add task state in profile #42895

[chore](profile) add task state in profile #42895

Uh oh!

Conversation

mrhhsg commented Oct 30, 2024

Proposed changes

Uh oh!

doris-robot commented Oct 30, 2024

Uh oh!

mrhhsg commented Oct 30, 2024

Uh oh!

github-actions bot commented Oct 30, 2024

Uh oh!

doris-robot commented Oct 30, 2024

Uh oh!

doris-robot commented Oct 30, 2024

Uh oh!

doris-robot commented Oct 30, 2024

Uh oh!

doris-robot commented Oct 30, 2024

Uh oh!

yiguolei Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiguolei Oct 30, 2024

Choose a reason for hiding this comment

Uh oh!

mrhhsg commented Oct 30, 2024

Uh oh!

github-actions bot commented Oct 30, 2024

Uh oh!

doris-robot commented Oct 30, 2024

Uh oh!

mrhhsg commented Oct 31, 2024

Uh oh!

github-actions bot commented Oct 31, 2024

Uh oh!

doris-robot commented Oct 31, 2024

Uh oh!

doris-robot commented Oct 31, 2024

Uh oh!

doris-robot commented Oct 31, 2024

Uh oh!

doris-robot commented Oct 31, 2024

Uh oh!

github-actions bot commented Oct 31, 2024

Uh oh!

github-actions bot commented Oct 31, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yiguolei Oct 30, 2024 •

edited

Loading