Skip to content

Conversation

@wangbo
Copy link
Contributor

@wangbo wangbo commented Mar 4, 2024

Proposed changes

Refactor active_queries() schema.

mysql [(none)]>desc function active_queries();
+------------------+--------+------+-------+---------+-------+
| Field            | Type   | Null | Key   | Default | Extra |
+------------------+--------+------+-------+---------+-------+
| QueryId          | TEXT   | No   | false | NULL    | NONE  |
| StartTime        | TEXT   | No   | false | NULL    | NONE  |
| QueryTimeMs      | BIGINT | No   | false | NULL    | NONE  |
| WorkloadGroupId  | BIGINT | No   | false | NULL    | NONE  |
| Database         | TEXT   | No   | false | NULL    | NONE  |
| FrontendInstance | TEXT   | No   | false | NULL    | NONE  |
| Sql              | TEXT   | No   | false | NULL    | NONE  |
+------------------+--------+------+-------+---------+-------+
7 rows in set (0.00 sec)

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@wangbo
Copy link
Contributor Author

wangbo commented Mar 4, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 4, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2024

PR approved by anyone and no changes requested.

@wangbo wangbo changed the title Refactor active_queries() schema [refactor](profile)Refactor active_queries() schema Mar 5, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 38013 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aebc87c9f7dc7839a0157a2ce49ab8bdf4268491, data reload: false

------ Round 1 ----------------------------------
q1	17630	4032	4032	4032
q2	2017	152	143	143
q3	10561	986	935	935
q4	4659	943	944	943
q5	7603	2947	3023	2947
q6	180	124	126	124
q7	1335	841	812	812
q8	9492	2098	2058	2058
q9	7266	6411	6417	6411
q10	8227	2559	2582	2559
q11	419	215	217	215
q12	742	315	313	313
q13	17974	2948	2917	2917
q14	288	245	257	245
q15	481	448	443	443
q16	469	408	394	394
q17	944	899	878	878
q18	6743	5926	5918	5918
q19	1586	1523	1505	1505
q20	564	281	295	281
q21	7546	3668	3648	3648
q22	791	299	292	292
Total cold run time: 107517 ms
Total hot run time: 38013 ms

----- Round 2, with runtime_filter_mode=off -----
q1	3971	3966	3968	3966
q2	318	231	229	229
q3	2944	2887	2921	2887
q4	1831	1839	1810	1810
q5	5231	5244	5217	5217
q6	201	123	118	118
q7	2257	1825	1830	1825
q8	3235	3256	3303	3256
q9	8474	8528	8483	8483
q10	6126	3770	3694	3694
q11	518	422	425	422
q12	684	574	528	528
q13	11124	2863	2738	2738
q14	274	246	247	246
q15	474	438	441	438
q16	454	392	406	392
q17	1691	1647	1667	1647
q18	7912	7408	7387	7387
q19	5105	1631	1608	1608
q20	1948	1763	1695	1695
q21	4971	4765	4792	4765
q22	470	399	469	399
Total cold run time: 70213 ms
Total hot run time: 53750 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176979 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aebc87c9f7dc7839a0157a2ce49ab8bdf4268491, data reload: false

query1	920	365	336	336
query2	7309	2074	2033	2033
query3	6702	219	210	210
query4	26956	20933	20653	20653
query5	4390	504	455	455
query6	271	178	181	178
query7	4626	315	301	301
query8	248	170	168	168
query9	8389	2176	2138	2138
query10	431	241	250	241
query11	14755	14344	14336	14336
query12	133	91	88	88
query13	1652	428	431	428
query14	9149	6733	7125	6733
query15	240	188	189	188
query16	7249	260	270	260
query17	927	558	550	550
query18	1906	281	286	281
query19	192	161	161	161
query20	96	95	91	91
query21	196	125	123	123
query22	4749	4499	4424	4424
query23	31577	30899	30954	30899
query24	11258	3010	3045	3010
query25	677	398	416	398
query26	1741	163	159	159
query27	2974	362	364	362
query28	6683	1822	1796	1796
query29	1157	596	610	596
query30	310	147	147	147
query31	937	726	736	726
query32	106	64	61	61
query33	766	272	260	260
query34	962	456	471	456
query35	916	797	777	777
query36	920	842	808	808
query37	142	66	66	66
query38	3208	3055	3040	3040
query39	1425	1363	1368	1363
query40	307	120	117	117
query41	60	57	56	56
query42	109	99	105	99
query43	427	410	401	401
query44	1076	730	716	716
query45	212	195	193	193
query46	1066	790	757	757
query47	1663	1559	1567	1559
query48	414	350	344	344
query49	1209	342	351	342
query50	777	376	373	373
query51	6530	6588	6467	6467
query52	112	92	96	92
query53	358	278	281	278
query54	286	238	246	238
query55	90	85	88	85
query56	247	234	232	232
query57	1073	1002	1000	1000
query58	260	228	218	218
query59	2413	2272	2329	2272
query60	271	273	244	244
query61	112	112	111	111
query62	653	424	416	416
query63	297	275	279	275
query64	6455	3186	3234	3186
query65	3033	2988	2973	2973
query66	1165	348	342	342
query67	15134	14551	14423	14423
query68	12981	564	592	564
query69	711	383	374	374
query70	1406	1125	1064	1064
query71	596	283	294	283
query72	9995	2637	2479	2479
query73	2343	332	342	332
query74	7180	6877	6843	6843
query75	5875	2612	2664	2612
query76	7097	1138	1180	1138
query77	663	250	250	250
query78	10407	9563	9500	9500
query79	10509	512	511	511
query80	1955	514	401	401
query81	488	209	208	208
query82	246	86	93	86
query83	245	148	139	139
query84	286	79	80	79
query85	1248	343	317	317
query86	362	273	292	273
query87	3448	3192	3165	3165
query88	3772	2375	2377	2375
query89	508	372	359	359
query90	2392	181	178	178
query91	160	129	128	128
query92	59	49	49	49
query93	5142	532	518	518
query94	1489	187	189	187
query95	456	364	356	356
query96	597	266	262	262
query97	4013	3900	3872	3872
query98	225	234	209	209
query99	1030	786	759	759
Total cold run time: 310938 ms
Total hot run time: 176979 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aebc87c9f7dc7839a0157a2ce49ab8bdf4268491, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.66	0.10	0.10
query5	0.52	0.49	0.50
query6	1.29	0.68	0.67
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.56	0.53	0.52
query10	0.58	0.56	0.58
query11	0.12	0.10	0.10
query12	0.13	0.10	0.10
query13	0.57	0.56	0.57
query14	0.76	0.75	0.76
query15	0.82	0.81	0.80
query16	0.37	0.37	0.36
query17	0.95	0.97	0.99
query18	0.25	0.25	0.25
query19	1.81	1.72	1.67
query20	0.02	0.01	0.02
query21	15.39	0.65	0.62
query22	2.92	4.53	2.02
query23	17.77	1.05	0.99
query24	2.15	0.57	0.12
query25	0.13	0.16	0.03
query26	0.16	0.14	0.14
query27	0.03	0.03	0.03
query28	12.50	0.83	0.82
query29	12.60	3.22	3.44
query30	0.62	0.54	0.58
query31	2.80	0.34	0.35
query32	3.36	0.44	0.44
query33	2.94	2.86	2.89
query34	15.51	4.33	4.30
query35	4.29	4.33	4.30
query36	1.08	1.02	1.01
query37	0.07	0.06	0.05
query38	0.05	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.15
query41	0.08	0.02	0.02
query42	0.03	0.02	0.01
query43	0.03	0.02	0.03
Total cold run time: 105.48 s
Total hot run time: 30.12 s

Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit aebc87c9f7dc7839a0157a2ce49ab8bdf4268491 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       18.3 seconds inserted 10000000 Rows, about 546K ops/s

@wangbo
Copy link
Contributor Author

wangbo commented Mar 5, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37547 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e0bc1a8d9ef654011b84627957bf61fcfd9f0a4, data reload: false

------ Round 1 ----------------------------------
q1	17701	4011	4008	4008
q2	2021	149	146	146
q3	10548	925	940	925
q4	4658	924	931	924
q5	7597	2795	2995	2795
q6	180	123	124	123
q7	1278	826	793	793
q8	9483	2034	2074	2034
q9	7310	6436	6399	6399
q10	8232	2578	2531	2531
q11	424	223	217	217
q12	746	308	312	308
q13	18010	2956	2908	2908
q14	274	251	247	247
q15	483	446	446	446
q16	484	388	420	388
q17	928	892	803	803
q18	6622	5835	5867	5835
q19	1571	1513	1502	1502
q20	551	281	270	270
q21	7489	3648	3671	3648
q22	809	297	310	297
Total cold run time: 107399 ms
Total hot run time: 37547 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4042	4015	4045	4015
q2	317	222	233	222
q3	2976	2895	2880	2880
q4	1836	1814	1791	1791
q5	5219	5280	5196	5196
q6	202	115	117	115
q7	2220	1852	1848	1848
q8	3210	3250	3232	3232
q9	8570	8466	8463	8463
q10	6139	3713	3705	3705
q11	530	438	439	438
q12	700	529	538	529
q13	12018	2792	2791	2791
q14	272	247	252	247
q15	461	447	441	441
q16	450	386	412	386
q17	1682	1622	1643	1622
q18	7745	7352	7215	7215
q19	5873	1584	1611	1584
q20	1947	1688	1710	1688
q21	4955	4713	4770	4713
q22	553	466	471	466
Total cold run time: 71917 ms
Total hot run time: 53587 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 177561 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4e0bc1a8d9ef654011b84627957bf61fcfd9f0a4, data reload: false

query1	943	359	347	347
query2	7330	2173	2056	2056
query3	6716	214	210	210
query4	26925	21010	21288	21010
query5	4320	601	4842	601
query6	277	183	175	175
query7	4620	312	319	312
query8	232	183	185	183
query9	8440	2176	2167	2167
query10	439	237	223	223
query11	14926	14342	14272	14272
query12	132	92	87	87
query13	1658	440	434	434
query14	8840	6855	6912	6855
query15	251	198	190	190
query16	7264	280	283	280
query17	942	599	572	572
query18	1945	293	303	293
query19	223	162	163	162
query20	94	94	95	94
query21	211	134	128	128
query22	4643	4478	4478	4478
query23	31392	30483	31106	30483
query24	12329	3034	3067	3034
query25	676	392	399	392
query26	1738	170	168	168
query27	3042	364	373	364
query28	6453	1804	1823	1804
query29	1153	624	629	624
query30	307	153	159	153
query31	933	725	768	725
query32	110	63	63	63
query33	776	266	262	262
query34	1015	468	490	468
query35	923	814	823	814
query36	950	823	830	823
query37	144	69	69	69
query38	3278	3084	3131	3084
query39	1427	1354	1385	1354
query40	299	124	121	121
query41	58	59	55	55
query42	113	104	104	104
query43	437	399	406	399
query44	1112	727	723	723
query45	207	200	195	195
query46	1053	798	773	773
query47	1643	1573	1575	1573
query48	428	351	363	351
query49	1152	346	340	340
query50	791	394	382	382
query51	6635	6560	6549	6549
query52	109	94	91	91
query53	359	285	284	284
query54	302	257	245	245
query55	90	85	84	84
query56	252	243	232	232
query57	1099	1022	982	982
query58	255	226	225	225
query59	2375	2292	2297	2292
query60	269	270	256	256
query61	135	115	117	115
query62	668	411	417	411
query63	304	277	278	277
query64	6490	3360	3104	3104
query65	3034	3000	3003	3000
query66	1159	333	323	323
query67	15164	14532	14299	14299
query68	12064	553	586	553
query69	716	379	383	379
query70	1376	1155	1094	1094
query71	587	292	276	276
query72	10085	2570	2515	2515
query73	2155	338	329	329
query74	7264	6781	6879	6781
query75	5943	2684	2704	2684
query76	7093	1075	1210	1075
query77	706	260	266	260
query78	10350	9647	9528	9528
query79	10635	520	534	520
query80	1620	474	381	381
query81	477	205	209	205
query82	317	94	90	90
query83	215	147	141	141
query84	281	77	83	77
query85	1193	335	330	330
query86	364	291	269	269
query87	3404	3217	3216	3216
query88	3526	2360	2355	2355
query89	527	365	364	364
query90	2033	174	178	174
query91	166	130	145	130
query92	56	50	50	50
query93	4618	524	504	504
query94	1309	187	188	187
query95	452	348	357	348
query96	587	271	271	271
query97	4037	3896	3900	3896
query98	232	214	204	204
query99	1075	777	799	777
Total cold run time: 309472 ms
Total hot run time: 177561 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.71 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4e0bc1a8d9ef654011b84627957bf61fcfd9f0a4, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.67	0.10	0.10
query5	0.48	0.48	0.49
query6	1.28	0.67	0.67
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.54	0.54
query10	0.57	0.56	0.56
query11	0.13	0.09	0.10
query12	0.12	0.10	0.11
query13	0.58	0.57	0.57
query14	0.75	0.74	0.76
query15	0.84	0.80	0.82
query16	0.37	0.37	0.38
query17	0.97	1.02	0.95
query18	0.27	0.24	0.28
query19	1.77	1.76	1.73
query20	0.02	0.01	0.01
query21	15.41	0.65	0.62
query22	3.36	3.15	2.28
query23	17.48	1.06	0.90
query24	2.11	0.39	0.58
query25	0.27	0.07	0.04
query26	0.16	0.13	0.14
query27	0.03	0.03	0.03
query28	12.16	0.85	0.84
query29	12.65	3.32	3.22
query30	0.63	0.56	0.58
query31	2.81	0.34	0.35
query32	3.36	0.43	0.44
query33	2.97	2.91	2.90
query34	15.48	4.32	4.30
query35	4.30	4.31	4.33
query36	1.12	1.01	1.02
query37	0.08	0.06	0.05
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.02	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.51 s
Total hot run time: 30.71 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4e0bc1a8d9ef654011b84627957bf61fcfd9f0a4 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       19.1 seconds inserted 10000000 Rows, about 523K ops/s

@wangbo wangbo merged commit 207aba4 into apache:master Mar 5, 2024
wangbo added a commit to wangbo/incubator-doris that referenced this pull request Mar 15, 2024
yiguolei pushed a commit that referenced this pull request Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed workload-group

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants