Skip to content

[feature](statistics)Support collect external partition stats. #34610

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Jibing-Li merged 1 commit into from
May 11, 2024
Merged

[feature](statistics)Support collect external partition stats. #34610

Jibing-Li merged 1 commit into from
May 11, 2024

Conversation

@Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented May 9, 2024 •
edited
Loading

This is the following pr of #34552. To support collect partition level stats for HMSExternalTable.
Also refactor analysis task code.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented May 9, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented May 9, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 9, 2024

TPC-H: Total hot run time: 41929 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7ede21e514143fa9ce44b9786ab3f0c3b6fc58f8, data reload: false

------ Round 1 ----------------------------------
q1	17609	4343	4221	4221
q2	1933	202	192	192
q3	7528	1334	1273	1273
q4	9804	833	844	833
q5	7611	2708	2769	2708
q6	221	134	135	134
q7	1053	613	619	613
q8	9272	2167	2123	2123
q9	7032	6665	6653	6653
q10	9082	3903	3879	3879
q11	457	244	250	244
q12	440	231	227	227
q13	16580	3172	3098	3098
q14	268	224	227	224
q15	512	473	457	457
q16	521	407	408	407
q17	1068	624	694	624
q18	8416	7726	7729	7726
q19	7263	1575	1535	1535
q20	679	322	327	322
q21	5233	4317	4160	4160
q22	349	276	276	276
Total cold run time: 112931 ms
Total hot run time: 41929 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4273	4190	4215	4190
q2	374	264	272	264
q3	2951	2772	2734	2734
q4	1879	1626	1620	1620
q5	5310	5312	5299	5299
q6	213	127	130	127
q7	2282	1882	1915	1882
q8	3217	3340	3325	3325
q9	8318	8347	8347	8347
q10	3841	3710	3662	3662
q11	576	496	506	496
q12	735	607	590	590
q13	16279	2984	2971	2971
q14	273	275	245	245
q15	505	469	472	469
q16	485	423	412	412
q17	1755	1469	1501	1469
q18	7691	7690	7379	7379
q19	1649	1517	1550	1517
q20	1937	1773	1736	1736
q21	9174	4741	4830	4741
q22	580	497	476	476
Total cold run time: 74297 ms
Total hot run time: 53951 ms

@doris-robot
Copy link

doris-robot commented May 9, 2024

TPC-DS: Total hot run time: 187780 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7ede21e514143fa9ce44b9786ab3f0c3b6fc58f8, data reload: false

query1	911	361	366	361
query2	6460	2360	2241	2241
query3	6647	205	210	205
query4	22882	21294	21149	21149
query5	4093	416	421	416
query6	267	171	189	171
query7	4574	298	294	294
query8	254	200	201	200
query9	8666	2425	2407	2407
query10	438	249	259	249
query11	14738	14147	14149	14147
query12	135	88	90	88
query13	1646	376	367	367
query14	9762	7765	7709	7709
query15	219	183	170	170
query16	7130	260	259	259
query17	1035	559	561	559
query18	1909	300	274	274
query19	203	157	149	149
query20	89	85	86	85
query21	196	128	132	128
query22	5076	4865	4861	4861
query23	34028	33520	33716	33520
query24	5088	2914	2940	2914
query25	481	385	380	380
query26	692	164	160	160
query27	1821	332	344	332
query28	3726	2074	2067	2067
query29	851	625	625	625
query30	226	156	164	156
query31	924	753	731	731
query32	61	54	54	54
query33	391	268	260	260
query34	867	493	495	493
query35	778	711	697	697
query36	1035	945	937	937
query37	106	68	71	68
query38	2884	2793	2774	2774
query39	1633	1556	1580	1556
query40	213	132	129	129
query41	44	41	44	41
query42	106	101	103	101
query43	558	540	563	540
query44	1057	726	748	726
query45	273	254	262	254
query46	1065	738	728	728
query47	1950	1868	1882	1868
query48	375	304	300	300
query49	781	407	413	407
query50	793	405	395	395
query51	6906	6866	6811	6811
query52	107	90	99	90
query53	349	296	291	291
query54	546	446	447	446
query55	78	74	75	74
query56	249	236	234	234
query57	1240	1127	1172	1127
query58	303	197	197	197
query59	3634	3325	3073	3073
query60	262	240	242	240
query61	90	88	91	88
query62	564	464	456	456
query63	305	285	286	285
query64	7689	7404	7376	7376
query65	3153	3063	3126	3063
query66	660	336	329	329
query67	15628	15319	15103	15103
query68	8592	545	550	545
query69	562	316	319	316
query70	1227	1139	1133	1133
query71	536	270	275	270
query72	8099	2535	2371	2371
query73	748	327	329	327
query74	6491	6223	6151	6151
query75	4476	2640	2608	2608
query76	4439	988	1011	988
query77	691	284	265	265
query78	10913	10189	10273	10189
query79	8144	518	514	514
query80	1118	434	438	434
query81	474	221	219	219
query82	692	100	91	91
query83	211	164	169	164
query84	266	89	92	89
query85	838	296	258	258
query86	375	310	306	306
query87	3311	3100	3129	3100
query88	4391	2429	2408	2408
query89	547	383	374	374
query90	2175	190	190	190
query91	123	98	99	98
query92	62	47	50	47
query93	6130	523	509	509
query94	947	186	186	186
query95	393	314	297	297
query96	603	274	270	270
query97	3119	2949	2972	2949
query98	255	220	207	207
query99	1249	925	922	922
Total cold run time: 284097 ms
Total hot run time: 187780 ms

@Jibing-Li Jibing-Li marked this pull request as ready for review May 10, 2024 06:49
@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented May 10, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented May 10, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 10, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 10, 2024

PR approved by anyone and no changes requested.

@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented May 10, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 10, 2024

TPC-H: Total hot run time: 41676 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 738f0167e5ed0b91b26db3806eb41db23306e1dc, data reload: false

------ Round 1 ----------------------------------
q1	17600	4271	4233	4233
q2	2007	181	195	181
q3	10457	1201	1157	1157
q4	10202	820	786	786
q5	7481	2727	2623	2623
q6	217	131	132	131
q7	1005	622	610	610
q8	9231	2147	2133	2133
q9	9808	6754	6768	6754
q10	9153	3907	3862	3862
q11	461	244	240	240
q12	436	213	238	213
q13	17221	3147	3177	3147
q14	261	233	227	227
q15	507	474	470	470
q16	496	374	367	367
q17	957	711	697	697
q18	8358	7879	7697	7697
q19	6676	1507	1526	1507
q20	624	330	320	320
q21	5167	4039	4254	4039
q22	356	282	290	282
Total cold run time: 118681 ms
Total hot run time: 41676 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4638	4385	4352	4352
q2	371	279	281	279
q3	3107	2890	2868	2868
q4	1863	1599	1571	1571
q5	5514	5502	5535	5502
q6	214	121	120	120
q7	2287	1970	1934	1934
q8	3247	3394	3389	3389
q9	8607	8700	8664	8664
q10	3942	3678	3847	3678
q11	581	488	486	486
q12	789	616	608	608
q13	16119	3153	3108	3108
q14	314	274	292	274
q15	541	485	490	485
q16	484	438	428	428
q17	1787	1460	1481	1460
q18	7724	7518	7428	7428
q19	1659	1577	1584	1577
q20	1956	1805	1739	1739
q21	9340	4797	4788	4788
q22	576	482	479	479
Total cold run time: 75660 ms
Total hot run time: 55217 ms

@doris-robot
Copy link

doris-robot commented May 10, 2024

TPC-DS: Total hot run time: 186872 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 738f0167e5ed0b91b26db3806eb41db23306e1dc, data reload: false

query1	901	364	352	352
query2	6435	2398	2414	2398
query3	6643	211	210	210
query4	23133	21206	21296	21206
query5	4137	418	405	405
query6	260	182	165	165
query7	4583	288	282	282
query8	252	204	195	195
query9	8500	2405	2391	2391
query10	429	246	255	246
query11	14763	14179	14231	14179
query12	135	89	88	88
query13	1652	395	364	364
query14	10489	7553	8265	7553
query15	230	172	176	172
query16	7945	272	268	268
query17	1768	571	559	559
query18	2043	289	280	280
query19	206	157	156	156
query20	93	88	88	88
query21	203	131	128	128
query22	5197	4949	4929	4929
query23	34226	33632	33575	33575
query24	10519	2907	2856	2856
query25	589	385	376	376
query26	694	168	154	154
query27	2209	307	323	307
query28	5675	2056	2040	2040
query29	888	638	628	628
query30	244	163	157	157
query31	1018	781	753	753
query32	100	60	56	56
query33	640	261	250	250
query34	873	470	490	470
query35	801	670	700	670
query36	1070	906	918	906
query37	101	69	71	69
query38	2892	2805	2748	2748
query39	1606	1542	1575	1542
query40	207	130	138	130
query41	46	41	43	41
query42	105	97	96	96
query43	616	556	538	538
query44	1097	713	704	704
query45	270	251	263	251
query46	1082	753	711	711
query47	2000	1907	1982	1907
query48	374	285	295	285
query49	834	387	393	387
query50	750	380	384	380
query51	6677	6561	6641	6561
query52	102	92	92	92
query53	349	277	273	273
query54	842	421	439	421
query55	77	72	73	72
query56	237	224	223	223
query57	1270	1161	1191	1161
query58	224	198	198	198
query59	3466	3221	3124	3124
query60	261	234	237	234
query61	91	90	89	89
query62	629	498	449	449
query63	298	272	280	272
query64	8403	7376	7322	7322
query65	3145	3133	3073	3073
query66	786	352	335	335
query67	16082	14966	15040	14966
query68	8873	550	540	540
query69	613	307	321	307
query70	1374	1119	1096	1096
query71	516	270	270	270
query72	8391	2525	2350	2350
query73	1627	321	357	321
query74	6637	6123	6136	6123
query75	4745	2634	2613	2613
query76	5152	997	952	952
query77	679	263	265	263
query78	10715	10062	9990	9990
query79	7654	502	513	502
query80	1192	447	434	434
query81	493	220	231	220
query82	671	95	90	90
query83	197	167	161	161
query84	267	91	85	85
query85	1081	271	261	261
query86	393	283	305	283
query87	3319	3113	3145	3113
query88	4584	2388	2393	2388
query89	511	380	378	378
query90	2040	192	200	192
query91	124	100	99	99
query92	56	51	50	50
query93	5579	512	501	501
query94	1157	179	176	176
query95	386	303	299	299
query96	613	267	268	267
query97	3161	3000	2970	2970
query98	239	226	215	215
query99	1206	897	899	897
Total cold run time: 298707 ms
Total hot run time: 186872 ms

morningman
morningman approved these changes May 11, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Jibing-Li Jibing-Li merged commit 96b2546 into apache:master May 11, 2024
@Jibing-Li Jibing-Li deleted the externalsql branch May 11, 2024 07:56
ByteYue pushed a commit to ByteYue/doris that referenced this pull request May 15, 2024
@Jibing-Li @ByteYue
Support collect external partition stats. (apache#34610)
2bd4fed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged not-merge/2.1 reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Jibing-Li @doris-robot @morningman @morrySnow @dataroaring