Skip to content

Conversation

@BePPPower
Copy link
Contributor

bp: #35515

Previously, irrelevant properties passed by users when using TVF were directly ignored by Doris.
Now, we retain and pass these additional user-defined properties to the S3 SDK.

Issue Number: close #xxx

Previously, irrelevant properties passed by users when using TVF were
directly ignored by Doris.
Now, we retain and pass these additional user-defined properties to the
S3 SDK.

<!--Describe your changes.-->

If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

TPC-H: Total hot run time: 50141 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 777663acebbface7258c5770a08faa2f190f582c, data reload: false

------ Round 1 ----------------------------------
q1	17867	4467	4414	4414
q2	2076	161	148	148
q3	10438	1929	1873	1873
q4	10352	1258	1318	1258
q5	8805	3962	3990	3962
q6	237	126	126	126
q7	2111	1594	1623	1594
q8	9283	2755	2722	2722
q9	11116	10750	10432	10432
q10	8666	3561	3543	3543
q11	420	249	255	249
q12	466	294	295	294
q13	18372	4002	4021	4002
q14	351	324	324	324
q15	514	462	464	462
q16	681	584	573	573
q17	1144	953	979	953
q18	7477	7022	6860	6860
q19	1684	1603	1535	1535
q20	521	317	311	311
q21	4493	4136	4111	4111
q22	491	396	395	395
Total cold run time: 117565 ms
Total hot run time: 50141 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4366	4336	4305	4305
q2	328	222	231	222
q3	4174	4169	4192	4169
q4	2755	2768	2756	2756
q5	7240	7206	7199	7199
q6	240	123	119	119
q7	3221	2892	2859	2859
q8	4376	4494	4504	4494
q9	17526	17076	17217	17076
q10	4239	4283	4319	4283
q11	755	707	689	689
q12	1044	854	878	854
q13	7295	3748	3743	3743
q14	447	431	421	421
q15	499	459	453	453
q16	764	682	677	677
q17	3826	3857	3780	3780
q18	8868	8875	8856	8856
q19	1723	1698	1644	1644
q20	2390	2150	2135	2135
q21	8605	8549	8597	8549
q22	1027	991	961	961
Total cold run time: 85708 ms
Total hot run time: 80244 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201397 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 777663acebbface7258c5770a08faa2f190f582c, data reload: false

query1	938	409	436	409
query2	6535	2307	2047	2047
query3	6929	209	200	200
query4	19953	17992	17884	17884
query5	19734	6495	6539	6495
query6	313	220	222	220
query7	4160	296	326	296
query8	295	259	238	238
query9	3160	2700	2655	2655
query10	428	346	305	305
query11	11344	10771	10848	10771
query12	121	77	80	77
query13	5610	664	676	664
query14	18206	13823	13176	13176
query15	390	229	242	229
query16	6451	291	271	271
query17	1726	1455	886	886
query18	2302	415	410	410
query19	215	154	150	150
query20	77	79	78	78
query21	195	99	93	93
query22	5203	5118	5163	5118
query23	32751	31889	31939	31889
query24	6904	6506	6446	6446
query25	526	464	428	428
query26	527	165	159	159
query27	1895	294	290	290
query28	6118	2331	2284	2284
query29	2846	2753	2964	2753
query30	242	169	171	169
query31	904	730	755	730
query32	71	64	64	64
query33	408	255	252	252
query34	853	474	484	474
query35	1141	967	946	946
query36	1363	1211	1172	1172
query37	86	65	60	60
query38	3111	3004	2962	2962
query39	1372	1342	1311	1311
query40	218	95	94	94
query41	39	37	37	37
query42	86	85	78	78
query43	628	607	528	528
query44	1141	720	735	720
query45	245	238	232	232
query46	1230	960	965	960
query47	1909	1846	1685	1685
query48	983	683	676	676
query49	619	380	366	366
query50	895	644	631	631
query51	4792	4634	4604	4604
query52	97	84	93	84
query53	457	321	328	321
query54	2638	2473	2470	2470
query55	81	80	82	80
query56	241	210	204	204
query57	1221	1220	1142	1142
query58	226	206	209	206
query59	3513	3350	3194	3194
query60	220	209	197	197
query61	102	101	103	101
query62	830	524	507	507
query63	486	333	341	333
query64	2355	1397	1446	1397
query65	3631	3587	3573	3573
query66	824	392	396	392
query67	15327	14992	14738	14738
query68	9641	647	653	647
query69	592	340	349	340
query70	1881	1373	1302	1302
query71	414	319	318	318
query72	6654	3544	3559	3544
query73	743	322	318	318
query74	6270	5902	5851	5851
query75	5355	3686	3685	3685
query76	6197	1152	1227	1152
query77	1036	267	263	263
query78	12820	11649	11859	11649
query79	6483	649	644	644
query80	941	409	416	409
query81	490	237	234	234
query82	1459	104	103	103
query83	164	135	130	130
query84	261	75	76	75
query85	897	339	321	321
query86	332	299	320	299
query87	3358	3072	3013	3013
query88	4690	2309	2323	2309
query89	379	309	284	284
query90	1959	214	214	214
query91	158	124	126	124
query92	58	52	51	51
query93	5161	583	562	562
query94	707	211	215	211
query95	1113	1061	1039	1039
query96	647	327	322	322
query97	6600	6374	6464	6374
query98	210	181	178	178
query99	2884	821	869	821
Total cold run time: 310991 ms
Total hot run time: 201397 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 777663acebbface7258c5770a08faa2f190f582c, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.02
query3	0.24	0.04	0.04
query4	1.79	0.06	0.07
query5	0.54	0.53	0.52
query6	1.24	0.66	0.63
query7	0.01	0.02	0.01
query8	0.04	0.03	0.02
query9	0.53	0.49	0.48
query10	0.54	0.54	0.53
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.64	0.62	0.62
query14	0.78	0.79	0.80
query15	0.77	0.76	0.75
query16	0.36	0.36	0.35
query17	1.02	1.03	1.01
query18	0.23	0.26	0.24
query19	1.91	1.86	1.82
query20	0.01	0.02	0.01
query21	15.44	0.61	0.53
query22	2.10	2.21	1.65
query23	17.07	1.04	0.94
query24	4.67	3.53	2.06
query25	0.42	0.06	0.06
query26	0.81	0.16	0.15
query27	0.04	0.05	0.04
query28	4.97	0.75	0.77
query29	12.61	2.21	2.29
query30	0.59	0.55	0.54
query31	2.82	0.38	0.37
query32	3.41	0.50	0.50
query33	3.09	3.09	3.08
query34	15.23	4.81	4.84
query35	4.88	4.87	4.86
query36	1.09	1.02	1.03
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.03	0.02	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.02
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 100.63 s
Total hot run time: 31.94 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 777663acebbface7258c5770a08faa2f190f582c with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@BePPPower
Copy link
Contributor Author

run p0

@morningman morningman merged commit ddcec43 into apache:branch-2.0 Aug 14, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
bp: apache#35515

Previously, irrelevant properties passed by users when using TVF were
directly ignored by Doris.
Now, we retain and pass these additional user-defined properties to the
S3 SDK.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants