Skip to content

branch-3.0: [opt](arrow-flight-sql) Support arrow-flight-sql protocol getStreamCatalogs, getStreamSchemas, getStreamTables #46217 #46269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
github-actions wants to merge 1 commit into from
Closed

branch-3.0: [opt](arrow-flight-sql) Support arrow-flight-sql protocol getStreamCatalogs, getStreamSchemas, getStreamTables #46217 #46269

github-actions wants to merge 1 commit into from

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jan 2, 2025

Cherry-picked from #46217

@xinyiZzz
[opt](arrow-flight-sql) Support arrow-flight-sql protocol `getStrea…
76bb8e5 
…mCatalogs`, `getStreamSchemas`, `getStreamTables` (#46217)

### What problem does this PR solve?

Implement the `getStreamCatalogs`, `getStreamSchemas`, `getStreamTables`
methods in the arrow-flight-sql protocol, which can support BI tools to
correctly display the metadata tree when using the `arrow-flight-sql`
Driver to connect to Doris.

DBeaver uses the `arrow-flight-sql` Driver connecting to Doris:

1. list all catalogs and show properties

![image](https://github.com/user-attachments/assets/f1de6e87-ba5d-4d67-a7bb-06fd91b6cbb1)
2. list dbSchemas and show properties

![image](https://github.com/user-attachments/assets/e706a065-420a-4137-a6a1-6e4f807fad8a)
3. list tables and list table columns.

![image](https://github.com/user-attachments/assets/f9929da9-8cc3-4d74-9f73-b3837854c349)
4. external catalog

![image](https://github.com/user-attachments/assets/ef9ebee1-36d8-4f7f-b97a-e4720ed45d1c)

![image](https://github.com/user-attachments/assets/58f1e0d9-17ed-48a9-be89-dfb818672525)

How to connect to Doris: (will be organized into documents later)

https://www.dremio.com/blog/jdbc-driver-for-arrow-flight-sql/#h-how-to-use-jdbc-driver-with-dbeaver-client

https://docs.dremio.com/current/sonar/client-applications/clients/dbeaver/?_gl=1*1epgwh0*_gcl_au*MjUyNjE1ODM0LjE3MzQwMDExNDg.
@Thearas
Copy link
Contributor

Thearas commented Jan 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Jan 2, 2025
@dataroaring dataroaring reopened this Jan 2, 2025
@Thearas
Copy link
Contributor

Thearas commented Jan 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

TPC-H: Total hot run time: 40900 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 76bb8e504727c0e4d969cbc217a5139d15a15b70, data reload: false

------ Round 1 ----------------------------------
q1	17589	7381	7268	7268
q2	2061	173	164	164
q3	10551	1118	1176	1118
q4	10219	758	753	753
q5	7752	2814	2817	2814
q6	240	148	147	147
q7	965	614	617	614
q8	9361	1932	2007	1932
q9	6639	6397	6373	6373
q10	7011	2327	2335	2327
q11	454	265	263	263
q12	413	215	217	215
q13	17787	3000	3021	3000
q14	240	209	205	205
q15	581	527	521	521
q16	668	598	584	584
q17	955	555	542	542
q18	7374	6674	6752	6674
q19	1395	1038	1071	1038
q20	478	204	209	204
q21	4006	3204	3160	3160
q22	1088	984	999	984
Total cold run time: 107827 ms
Total hot run time: 40900 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7271	7218	7216	7216
q2	326	245	225	225
q3	2891	3022	2938	2938
q4	2017	1816	1838	1816
q5	5714	5711	5756	5711
q6	214	137	138	137
q7	2140	1812	1820	1812
q8	3336	3523	3477	3477
q9	8757	8781	8812	8781
q10	3602	3501	3571	3501
q11	599	502	504	502
q12	817	629	596	596
q13	9821	3099	3110	3099
q14	327	269	282	269
q15	580	524	513	513
q16	704	665	665	665
q17	1756	1573	1545	1545
q18	7762	7547	7356	7356
q19	1642	1581	1575	1575
q20	2018	1823	1820	1820
q21	5354	5239	5171	5171
q22	1089	1027	1017	1017
Total cold run time: 68737 ms
Total hot run time: 59742 ms

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

TPC-DS: Total hot run time: 192237 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 76bb8e504727c0e4d969cbc217a5139d15a15b70, data reload: false

query1	978	366	367	366
query2	6532	2076	2051	2051
query3	6711	216	218	216
query4	41014	23633	23834	23633
query5	5286	470	465	465
query6	245	184	181	181
query7	4085	308	311	308
query8	294	237	245	237
query9	6497	2715	2706	2706
query10	458	268	259	259
query11	16101	15102	15164	15102
query12	152	104	108	104
query13	1002	424	423	423
query14	9809	7536	7387	7387
query15	212	176	173	173
query16	7760	460	474	460
query17	1603	567	552	552
query18	2009	308	303	303
query19	385	151	150	150
query20	117	113	106	106
query21	64	46	44	44
query22	4629	4151	4213	4151
query23	34761	34456	34079	34079
query24	11731	2918	2906	2906
query25	630	398	414	398
query26	1833	171	171	171
query27	2821	295	319	295
query28	7984	2454	2410	2410
query29	904	458	437	437
query30	329	171	174	171
query31	1003	808	822	808
query32	67	58	63	58
query33	635	312	312	312
query34	918	508	519	508
query35	840	728	753	728
query36	1084	966	947	947
query37	181	76	80	76
query38	4026	3973	3799	3799
query39	1512	1439	1429	1429
query40	224	88	85	85
query41	56	49	49	49
query42	116	105	104	104
query43	539	483	489	483
query44	1315	823	812	812
query45	188	170	176	170
query46	1158	725	724	724
query47	1902	1823	1828	1823
query48	463	379	381	379
query49	1237	424	414	414
query50	809	413	420	413
query51	7265	7278	7202	7202
query52	101	94	90	90
query53	272	189	189	189
query54	589	470	468	468
query55	80	81	79	79
query56	274	267	258	258
query57	1263	1125	1125	1125
query58	242	214	217	214
query59	3112	2865	2888	2865
query60	301	275	270	270
query61	136	133	132	132
query62	841	683	689	683
query63	221	197	195	195
query64	4757	759	725	725
query65	3314	3167	3178	3167
query66	1102	309	313	309
query67	15850	15597	15441	15441
query68	4268	575	573	573
query69	421	268	267	267
query70	1194	1101	1063	1063
query71	328	251	261	251
query72	6521	4094	4051	4051
query73	777	346	354	346
query74	9981	9088	8928	8928
query75	3348	2607	2644	2607
query76	2225	986	1112	986
query77	435	285	294	285
query78	10673	9558	9694	9558
query79	1394	630	627	627
query80	930	450	437	437
query81	544	242	239	239
query82	1228	120	122	120
query83	211	143	140	140
query84	237	80	84	80
query85	1287	318	302	302
query86	371	296	312	296
query87	4510	4305	4317	4305
query88	3388	2405	2362	2362
query89	399	289	297	289
query90	1835	191	192	191
query91	186	168	149	149
query92	59	52	50	50
query93	1067	551	548	548
query94	708	309	269	269
query95	350	262	257	257
query96	608	276	286	276
query97	3382	3165	3116	3116
query98	215	209	196	196
query99	1525	1296	1324	1296
Total cold run time: 299534 ms
Total hot run time: 192237 ms

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

ClickBench: Total hot run time: 33.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 76bb8e504727c0e4d969cbc217a5139d15a15b70, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.10
query5	0.54	0.49	0.52
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.51	0.48
query10	0.54	0.53	0.56
query11	0.15	0.10	0.10
query12	0.13	0.11	0.11
query13	0.60	0.60	0.59
query14	3.07	2.93	2.97
query15	0.92	0.82	0.82
query16	0.40	0.37	0.41
query17	1.05	0.98	1.03
query18	0.23	0.21	0.22
query19	2.03	1.97	1.77
query20	0.01	0.01	0.01
query21	15.36	0.58	0.58
query22	2.26	2.35	2.54
query23	16.80	0.90	0.95
query24	2.72	1.15	1.49
query25	0.24	0.13	0.21
query26	0.39	0.14	0.14
query27	0.05	0.04	0.05
query28	10.35	1.10	1.06
query29	12.54	3.22	3.22
query30	0.24	0.07	0.07
query31	2.84	0.39	0.38
query32	3.26	0.46	0.46
query33	2.99	2.99	3.07
query34	16.91	4.43	4.49
query35	4.46	4.47	4.49
query36	0.67	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.12
query41	0.07	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.97 s
Total hot run time: 33.52 s

@CalvinKirs CalvinKirs deleted the auto-pick-46217-branch-3.0 branch March 28, 2025 06:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @xinyiZzz