Skip to content

[feature](paimon)support paimon with dlf for 3.0 (#41247) #41695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Oct 12, 2024
Merged

[feature](paimon)support paimon with dlf for 3.0 (#41247) #41695

morningman merged 1 commit into from
Oct 12, 2024

Conversation

@wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Oct 11, 2024

bp: #41247

@wuwenchi
[feature](paimon)support paimon with dlf (apache#41247)
2a327ef 
## Proposed changes

We now support reading the paimon table on dlf. 
We can create a catalog using dlf for paimon in the following way:

```
CREATE CATALOG `dlf_paimon` PROPERTIES (
"type" = "paimon",
"paimon.catalog.type" = "dlf",
"warehouse" = "oss://xx/yy/",
"dlf.proxy.mode" = "DLF_ONLY",
"dlf.uid" = "xxxxx",
"dlf.region" = "cn-beijing",
"dlf.access_key" = "ak",
"dlf.secret_key" = "sk"

-- "dlf.endpoint" = "dlf.cn-beijing.aliyuncs.com",  -- optional
-- "dlf.catalog.id" = "xxxx", -- optional
);
```
@doris-robot
Copy link

doris-robot commented Oct 11, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Oct 11, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 11, 2024

TPC-H: Total hot run time: 40155 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2a327efb671a079c5a348502d880e5e26b669d54, data reload: false

------ Round 1 ----------------------------------
q1	17895	7426	7266	7266
q2	2037	157	192	157
q3	10823	1035	1138	1035
q4	10555	737	730	730
q5	7768	2810	2843	2810
q6	224	151	146	146
q7	993	644	627	627
q8	9371	1916	1949	1916
q9	6617	6438	6363	6363
q10	7005	2252	2269	2252
q11	430	245	244	244
q12	411	214	212	212
q13	17779	2955	2940	2940
q14	239	211	208	208
q15	561	510	520	510
q16	497	398	413	398
q17	980	595	504	504
q18	7292	6635	6488	6488
q19	1768	924	961	924
q20	560	277	269	269
q21	3888	3267	3173	3173
q22	1102	1003	983	983
Total cold run time: 108795 ms
Total hot run time: 40155 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7303	7245	7228	7228
q2	327	223	225	223
q3	2837	2861	2889	2861
q4	2035	1818	1864	1818
q5	5702	5771	5707	5707
q6	224	137	140	137
q7	2195	1741	1754	1741
q8	3307	3492	3482	3482
q9	8777	8842	8867	8842
q10	3535	3499	3536	3499
q11	585	479	467	467
q12	813	631	602	602
q13	16483	3128	3131	3128
q14	300	268	288	268
q15	561	529	531	529
q16	540	473	445	445
q17	1878	1649	1622	1622
q18	8197	7701	7709	7701
q19	2368	1593	1357	1357
q20	2113	1848	1849	1848
q21	5470	5305	5226	5226
q22	1148	1012	1024	1012
Total cold run time: 76698 ms
Total hot run time: 59743 ms

@doris-robot
Copy link

doris-robot commented Oct 11, 2024

TPC-DS: Total hot run time: 193646 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2a327efb671a079c5a348502d880e5e26b669d54, data reload: false

query1	1298	901	894	894
query2	6260	1972	1987	1972
query3	10918	4004	3824	3824
query4	66986	24788	23506	23506
query5	5892	476	449	449
query6	493	158	157	157
query7	6458	295	293	293
query8	293	200	200	200
query9	9429	2678	2656	2656
query10	520	284	275	275
query11	18415	15254	15862	15254
query12	165	99	103	99
query13	1686	465	433	433
query14	11099	7079	7277	7079
query15	218	175	170	170
query16	7315	532	515	515
query17	1077	585	575	575
query18	1889	330	309	309
query19	187	143	138	138
query20	113	107	106	106
query21	216	98	98	98
query22	4583	4589	4484	4484
query23	34228	33615	33781	33615
query24	5559	2924	2871	2871
query25	527	397	395	395
query26	688	163	158	158
query27	1715	292	300	292
query28	4048	2173	2169	2169
query29	649	439	432	432
query30	227	153	148	148
query31	1018	777	788	777
query32	70	52	53	52
query33	471	306	287	287
query34	866	500	486	486
query35	865	728	709	709
query36	1076	935	914	914
query37	133	76	78	76
query38	3984	3875	3806	3806
query39	1471	1417	1422	1417
query40	199	99	100	99
query41	48	44	45	44
query42	117	97	95	95
query43	517	479	479	479
query44	1133	751	767	751
query45	193	163	167	163
query46	1126	709	699	699
query47	1912	1863	1861	1861
query48	433	333	326	326
query49	706	378	387	378
query50	835	400	405	400
query51	7010	6899	6853	6853
query52	100	93	87	87
query53	260	185	189	185
query54	575	470	462	462
query55	74	76	77	76
query56	284	249	249	249
query57	1285	1123	1135	1123
query58	216	224	243	224
query59	3110	2881	3074	2881
query60	283	264	262	262
query61	98	100	96	96
query62	795	666	631	631
query63	225	185	182	182
query64	1700	604	586	586
query65	3250	3185	3194	3185
query66	746	299	299	299
query67	15929	15300	15299	15299
query68	3701	533	527	527
query69	617	281	296	281
query70	1134	1124	1099	1099
query71	512	262	268	262
query72	7942	3779	3838	3779
query73	765	336	342	336
query74	10308	8978	9129	8978
query75	4595	2641	2633	2633
query76	3295	890	864	864
query77	760	297	282	282
query78	9947	9693	9331	9331
query79	3727	570	561	561
query80	1127	429	424	424
query81	565	228	225	225
query82	681	128	129	128
query83	383	146	135	135
query84	297	77	86	77
query85	1684	291	277	277
query86	442	301	295	295
query87	4489	4317	4214	4214
query88	4850	2403	2423	2403
query89	404	279	289	279
query90	2067	178	185	178
query91	147	101	105	101
query92	62	44	46	44
query93	4450	522	514	514
query94	1016	281	292	281
query95	341	247	246	246
query96	621	272	275	272
query97	3332	3097	3119	3097
query98	218	198	198	198
query99	1721	1282	1260	1260
Total cold run time: 332464 ms
Total hot run time: 193646 ms

@morningman morningman merged commit 8c89217 into apache:branch-3.0 Oct 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wuwenchi @doris-robot @morningman