[Improvement](executor)Add queue columns for active_queries #32259

wangbo · 2024-03-14T11:28:09Z

Add QUEUE_START_TIME/QUEUE_END_TIME/QUERY_STATUS column for active_queries;

doris-robot · 2024-03-14T11:28:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

wangbo · 2024-03-14T11:28:18Z

run buildall

wangbo · 2024-03-14T11:30:33Z

run buildall

github-actions · 2024-03-14T11:34:11Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-14T11:35:33Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-03-14T11:52:58Z

TPC-H: Total hot run time: 38312 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0ea8ee4d4a2b18bd703d6f6c59a5becbb6ce2311, data reload: false

------ Round 1 ----------------------------------
q1	17705	4368	4129	4129
q2	2029	151	146	146
q3	10685	1120	912	912
q4	7813	785	796	785
q5	7507	2594	2715	2594
q6	185	121	122	121
q7	1223	833	814	814
q8	9380	2073	1983	1983
q9	7104	6451	6457	6451
q10	8429	3463	3621	3463
q11	424	230	225	225
q12	651	301	290	290
q13	17810	2838	2860	2838
q14	269	246	245	245
q15	494	457	456	456
q16	491	396	380	380
q17	949	548	544	544
q18	7385	6493	6481	6481
q19	4542	1406	1482	1406
q20	555	286	278	278
q21	6284	3478	3484	3478
q22	359	293	348	293
Total cold run time: 112273 ms
Total hot run time: 38312 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4171	4122	4051	4051
q2	324	231	228	228
q3	2949	2866	2830	2830
q4	1880	1565	1586	1565
q5	5216	5282	5264	5264
q6	197	123	114	114
q7	2251	1848	1825	1825
q8	3163	3283	3286	3283
q9	8604	8618	8600	8600
q10	3739	3676	3715	3676
q11	541	440	441	440
q12	725	568	518	518
q13	16913	2865	2851	2851
q14	272	244	256	244
q15	484	443	441	441
q16	452	408	400	400
q17	1729	1479	1473	1473
q18	7566	7270	7032	7032
q19	1632	1563	1574	1563
q20	1905	1703	1733	1703
q21	4829	4735	4766	4735
q22	530	473	425	425
Total cold run time: 70072 ms
Total hot run time: 53261 ms

doris-robot · 2024-03-14T11:56:57Z

TeamCity be ut coverage result:
Function Coverage: 34.99% (8573/24499)
Line Coverage: 26.73% (69458/259807)
Region Coverage: 26.01% (36079/138691)
Branch Coverage: 22.97% (18423/80208)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0ea8ee4d4a2b18bd703d6f6c59a5becbb6ce2311_0ea8ee4d4a2b18bd703d6f6c59a5becbb6ce2311/report/index.html

yiguolei · 2024-03-14T13:56:38Z

fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java

-                        LOG.error("query (id=" + DebugUtil.printId(queryId) + ") " + queueToken.getOfferResultDetail());
-                        queryQueue.returnToken(queueToken);
-                        throw new UserException(queueToken.getOfferResultDetail());
+                    long queueStartTime = System.currentTimeMillis();


这么改不好，容易锁死。
直接把queue start time和queue end time 放到queue token 里，然后token 本身是coordinator的一个属性就ok了。
当我们显示的时候，直接从token 里拿start time和end time 就可以了

wangbo · 2024-03-15T04:13:12Z

run buildall

github-actions · 2024-03-15T04:18:09Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2024-03-15T04:19:18Z

fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java

    }

+    @Override
+    public QueryInfo getQueryInfoByQueryId(TUniqueId queryId) throws UserException {


Both query start time and token are in coordinator, we could get these info directly from coordinator.
And QueryInfo contains a coordinator. So that we do not need these API any more.

yiguolei · 2024-03-15T04:21:06Z

fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java

            }
            trow.addToColumnValue(new TCell().setStringVal(selfNode));
+
+            QueueToken queueToken = queryInfo.getQueueToken();


Do not use token directly here.
Add related method in QueryInfo, for example getQueueStartTime, getQueueEndTime
and then QueryInfo could use coordinator to get these time.

yiguolei · 2024-03-15T04:22:05Z

fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java

            return startExecTime;
        }
+
+        public void setQueueToken(QueueToken queueToken) {


remove setQueueToken and getQueueToken setStartExecTime
Get these info from coordinator directly.

doris-robot · 2024-03-15T05:22:18Z

TPC-H: Total hot run time: 38578 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1d1c8f8996f40604d3b1943c5eea236b7fd24fa0, data reload: false

------ Round 1 ----------------------------------
q1	17642	4254	4159	4159
q2	2021	155	147	147
q3	10592	1098	905	905
q4	5001	729	750	729
q5	7413	2828	2814	2814
q6	200	120	127	120
q7	1167	817	825	817
q8	9342	2016	2018	2016
q9	7045	6496	6442	6442
q10	8553	3514	3655	3514
q11	440	218	217	217
q12	777	307	293	293
q13	17820	2850	2835	2835
q14	276	249	245	245
q15	503	452	444	444
q16	499	389	383	383
q17	957	550	641	550
q18	7328	6555	6404	6404
q19	3923	1425	1391	1391
q20	537	274	273	273
q21	7073	3570	3666	3570
q22	359	310	311	310
Total cold run time: 109468 ms
Total hot run time: 38578 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4143	4066	4081	4066
q2	316	229	220	220
q3	2946	2832	2870	2832
q4	1840	1569	1544	1544
q5	5273	5263	5253	5253
q6	200	117	116	116
q7	2190	1853	1815	1815
q8	3176	3304	3291	3291
q9	8532	8575	8616	8575
q10	3740	3693	3662	3662
q11	550	457	443	443
q12	728	538	549	538
q13	16930	2854	2845	2845
q14	273	243	246	243
q15	485	445	454	445
q16	440	424	402	402
q17	1730	1503	1464	1464
q18	7476	7296	7070	7070
q19	1619	1556	1557	1556
q20	1919	1697	1740	1697
q21	4762	4774	4710	4710
q22	543	455	482	455
Total cold run time: 69811 ms
Total hot run time: 53242 ms

doris-robot · 2024-03-15T05:35:50Z

TeamCity be ut coverage result:
Function Coverage: 34.95% (8574/24534)
Line Coverage: 26.64% (69470/260727)
Region Coverage: 25.94% (36086/139116)
Branch Coverage: 22.89% (18425/80484)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1d1c8f8996f40604d3b1943c5eea236b7fd24fa0_1d1c8f8996f40604d3b1943c5eea236b7fd24fa0/report/index.html

github-actions · 2024-03-15T06:26:38Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-15T06:40:04Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-15T06:42:41Z

clang-tidy review says "All clean, LGTM! 👍"

…eries

wangbo · 2024-03-15T06:46:21Z

run buildall

github-actions · 2024-03-15T06:49:12Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-15T06:51:55Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei

LGTM

github-actions · 2024-03-15T06:54:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-15T06:54:49Z

PR approved by anyone and no changes requested.

doris-robot · 2024-03-15T07:27:53Z

TPC-H: Total hot run time: 37997 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94af99aa45e4beab758ed3b318f13d439a6041f6, data reload: false

------ Round 1 ----------------------------------
q1	17626	4176	4097	4097
q2	2023	153	142	142
q3	10613	1052	896	896
q4	7086	706	692	692
q5	7448	2611	2679	2611
q6	181	121	120	120
q7	1182	789	794	789
q8	9339	1921	2004	1921
q9	7150	6387	6396	6387
q10	8495	3527	3640	3527
q11	438	225	211	211
q12	769	305	283	283
q13	18139	2810	2821	2810
q14	274	244	252	244
q15	497	462	442	442
q16	509	382	383	382
q17	945	557	512	512
q18	7004	6396	6509	6396
q19	1567	1388	1372	1372
q20	564	285	278	278
q21	6303	3594	3663	3594
q22	357	291	333	291
Total cold run time: 108509 ms
Total hot run time: 37997 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4090	4194	4103	4103
q2	318	226	229	226
q3	2928	2798	2792	2792
q4	1839	1579	1530	1530
q5	5230	5240	5210	5210
q6	190	116	116	116
q7	2210	1874	1844	1844
q8	3163	3287	3280	3280
q9	8536	8486	8504	8486
q10	3652	3707	3662	3662
q11	540	443	458	443
q12	708	553	559	553
q13	17211	2887	2979	2887
q14	289	268	264	264
q15	482	448	449	448
q16	474	445	423	423
q17	1747	1478	1459	1459
q18	7700	7150	7031	7031
q19	1675	1459	1451	1451
q20	1890	1670	1695	1670
q21	4868	4634	4621	4621
q22	539	449	436	436
Total cold run time: 70279 ms
Total hot run time: 52935 ms

doris-robot · 2024-03-15T07:40:42Z

TeamCity be ut coverage result:
Function Coverage: 34.95% (8574/24534)
Line Coverage: 26.65% (69485/260731)
Region Coverage: 25.94% (36087/139107)
Branch Coverage: 22.90% (18427/80478)
Coverage Report: http://coverage.selectdb-in.cc/coverage/94af99aa45e4beab758ed3b318f13d439a6041f6_94af99aa45e4beab758ed3b318f13d439a6041f6/report/index.html

morningman

LGTM

…eries (apache#32259)

wangbo added the workload-group label Mar 14, 2024

wangbo force-pushed the 0314_queue_info branch from 8ce1ce5 to 0ea8ee4 Compare March 14, 2024 11:30

yiguolei reviewed Mar 14, 2024

View reviewed changes

wangbo force-pushed the 0314_queue_info branch from 0ea8ee4 to 1d1c8f8 Compare March 15, 2024 04:12

yiguolei reviewed Mar 15, 2024

View reviewed changes

wangbo force-pushed the 0314_queue_info branch from 1d1c8f8 to 11d685b Compare March 15, 2024 06:20

wangbo force-pushed the 0314_queue_info branch from f5ebf9f to c6ef27c Compare March 15, 2024 06:44

Add QUEUE_START_TIME/QUEUE_END_TIME/QUERY_STATUS column for active_qu…

94af99a

…eries

wangbo force-pushed the 0314_queue_info branch from c6ef27c to 94af99a Compare March 15, 2024 06:46

yiguolei approved these changes Mar 15, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 15, 2024

morningman approved these changes Mar 15, 2024

View reviewed changes

yiguolei merged commit b1d6340 into apache:master Mar 15, 2024

yiguolei pushed a commit to yiguolei/incubator-doris that referenced this pull request Mar 16, 2024

Add QUEUE_START_TIME/QUEUE_END_TIME/QUERY_STATUS column for active_qu…

83ab61a

…eries (apache#32259)

yiguolei mentioned this pull request Mar 24, 2024

Release notes 2.1.1 #32736

Closed

[Improvement](executor)Add queue columns for active_queries #32259

[Improvement](executor)Add queue columns for active_queries #32259

Uh oh!

Conversation

wangbo commented Mar 14, 2024

Uh oh!

doris-robot commented Mar 14, 2024

Uh oh!

wangbo commented Mar 14, 2024

Uh oh!

wangbo commented Mar 14, 2024

Uh oh!

github-actions bot commented Mar 14, 2024

Uh oh!

github-actions bot commented Mar 14, 2024

Uh oh!

doris-robot commented Mar 14, 2024

Uh oh!

doris-robot commented Mar 14, 2024

Uh oh!

yiguolei Mar 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangbo commented Mar 15, 2024

Uh oh!

github-actions bot commented Mar 15, 2024

Uh oh!

yiguolei Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

yiguolei Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

yiguolei Mar 15, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Mar 15, 2024

Uh oh!

doris-robot commented Mar 15, 2024

Uh oh!

github-actions bot commented Mar 15, 2024

Uh oh!

github-actions bot commented Mar 15, 2024

Uh oh!

github-actions bot commented Mar 15, 2024

Uh oh!

wangbo commented Mar 15, 2024

Uh oh!

github-actions bot commented Mar 15, 2024

Uh oh!

github-actions bot commented Mar 15, 2024

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 15, 2024

Uh oh!

github-actions bot commented Mar 15, 2024

Uh oh!

doris-robot commented Mar 15, 2024

Uh oh!

doris-robot commented Mar 15, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yiguolei Mar 14, 2024 •

edited

Loading