Skip to content

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Feb 19, 2024

Proposed changes

the ranger-plugins-common depends aws-java-sdk-bundle, which is too large and unnecessary.
exclude it and add aws-java-sdk-logs alone.

The FE lib dir reduce size from 1.1GB to 713MB.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41299 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa88d09a26040c2a42ea3bba89e9f2c38db65d9b, data reload: false

------ Round 1 ----------------------------------
q1	17701	4927	4861	4861
q2	2041	143	131	131
q3	10580	1034	994	994
q4	4649	964	971	964
q5	7693	3209	3260	3209
q6	194	134	135	134
q7	1257	788	761	761
q8	9228	2063	2069	2063
q9	7618	6686	6689	6686
q10	8310	2652	2644	2644
q11	405	220	224	220
q12	729	336	338	336
q13	18001	3680	3670	3670
q14	288	259	255	255
q15	590	518	509	509
q16	491	408	408	408
q17	931	876	850	850
q18	7387	6756	6521	6521
q19	1539	1486	1485	1485
q20	659	324	344	324
q21	6481	3931	3946	3931
q22	888	358	343	343
Total cold run time: 107660 ms
Total hot run time: 41299 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4895	4921	4947	4921
q2	314	198	197	197
q3	3610	3614	3627	3614
q4	2547	2565	2554	2554
q5	5748	5745	5746	5745
q6	210	128	130	128
q7	2282	1664	1697	1664
q8	3076	3136	3200	3136
q9	8705	8699	8681	8681
q10	6793	4253	4226	4226
q11	573	424	432	424
q12	798	559	559	559
q13	4342	3408	3423	3408
q14	264	248	254	248
q15	607	510	529	510
q16	509	451	442	442
q17	1825	1767	1787	1767
q18	8454	7908	7819	7819
q19	1626	1625	1628	1625
q20	2127	1843	1853	1843
q21	6735	6405	6317	6317
q22	609	530	537	530
Total cold run time: 66649 ms
Total hot run time: 60358 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 178981 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa88d09a26040c2a42ea3bba89e9f2c38db65d9b, data reload: false

query1	915	355	349	349
query2	6536	1761	1658	1658
query3	6691	206	201	201
query4	23116	21177	21325	21177
query5	4231	391	387	387
query6	254	171	169	169
query7	4607	294	292	292
query8	255	213	202	202
query9	8821	2880	2863	2863
query10	416	224	225	224
query11	15153	14779	14647	14647
query12	162	84	84	84
query13	1703	435	426	426
query14	10167	8462	8817	8462
query15	790	212	217	212
query16	9787	314	289	289
query17	5266	1379	557	557
query18	3645	287	291	287
query19	981	154	168	154
query20	359	85	89	85
query21	908	129	140	129
query22	4916	4705	4698	4698
query23	32452	31607	31501	31501
query24	11535	3405	3508	3405
query25	627	372	382	372
query26	1570	156	161	156
query27	3036	311	317	311
query28	6495	1867	1853	1853
query29	1096	623	621	621
query30	275	138	145	138
query31	921	757	760	757
query32	95	60	56	56
query33	749	240	250	240
query34	1045	491	502	491
query35	948	855	849	849
query36	979	881	876	876
query37	129	60	65	60
query38	3252	3208	3143	3143
query39	1372	1342	1322	1322
query40	278	105	106	105
query41	37	35	35	35
query42	105	100	100	100
query43	468	441	446	441
query44	1059	698	717	698
query45	201	189	175	175
query46	1057	773	751	751
query47	1599	1551	1573	1551
query48	416	338	344	338
query49	1095	308	305	305
query50	783	378	372	372
query51	5295	5149	5219	5149
query52	107	99	92	92
query53	393	305	293	293
query54	308	225	234	225
query55	84	86	77	77
query56	227	204	198	198
query57	1075	935	954	935
query58	220	206	202	202
query59	2279	2223	2198	2198
query60	252	225	230	225
query61	92	83	82	82
query62	582	365	370	365
query63	337	298	287	287
query64	6306	3067	3149	3067
query65	3303	3245	3258	3245
query66	1351	330	321	321
query67	14680	14346	14240	14240
query68	5136	561	567	561
query69	529	370	360	360
query70	1226	1290	1255	1255
query71	443	285	257	257
query72	6343	2804	2640	2640
query73	707	313	310	310
query74	6909	6429	6358	6358
query75	3227	2580	2567	2567
query76	3290	1102	1215	1102
query77	370	239	249	239
query78	9452	8867	8740	8740
query79	966	515	523	515
query80	507	353	347	347
query81	439	217	209	209
query82	165	92	90	90
query83	143	128	130	128
query84	231	83	79	79
query85	1033	354	343	343
query86	304	308	312	308
query87	3439	3322	3325	3322
query88	2733	2274	2268	2268
query89	452	369	368	368
query90	2000	171	169	169
query91	159	130	132	130
query92	54	53	48	48
query93	1021	500	513	500
query94	1307	186	178	178
query95	8639	375	8486	375
query96	588	269	264	264
query97	4433	4259	4268	4259
query98	221	211	200	200
query99	1068	743	717	717
Total cold run time: 287778 ms
Total hot run time: 178981 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fa88d09a26040c2a42ea3bba89e9f2c38db65d9b, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.02	0.02
query3	0.23	0.07	0.08
query4	1.64	0.10	0.08
query5	0.49	0.48	0.48
query6	1.36	0.60	0.62
query7	0.02	0.02	0.01
query8	0.04	0.02	0.02
query9	0.54	0.46	0.46
query10	0.49	0.50	0.51
query11	0.12	0.10	0.10
query12	0.12	0.10	0.11
query13	0.60	0.58	0.59
query14	0.76	0.80	0.78
query15	0.82	0.81	0.80
query16	0.33	0.36	0.35
query17	0.91	0.87	0.93
query18	0.20	0.18	0.17
query19	1.77	1.63	1.67
query20	0.02	0.01	0.02
query21	15.41	0.66	0.58
query22	3.67	3.97	2.41
query23	17.37	1.07	1.01
query24	2.05	0.71	0.20
query25	0.64	0.05	0.07
query26	0.16	0.14	0.16
query27	0.06	0.06	0.05
query28	11.95	0.85	0.84
query29	12.59	3.35	3.32
query30	0.51	0.50	0.44
query31	2.79	0.37	0.38
query32	3.33	0.48	0.48
query33	3.14	3.14	3.17
query34	15.36	4.54	4.51
query35	4.50	4.53	4.52
query36	1.09	0.96	0.96
query37	0.07	0.05	0.06
query38	0.04	0.03	0.04
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.03	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.6 s
Total hot run time: 30.94 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit fa88d09a26040c2a42ea3bba89e9f2c38db65d9b with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 20, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 343346c into apache:master Feb 20, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.5-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants