Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 115 additions & 0 deletions docs/en/docs/ecosystem/kyuubi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
---

{
"title": "Kyuubi",
"language": "en"
}

---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Use Kyuubi with Doris

## Introduction

[Apache Kyuubi](https://kyuubi.apache.org/) is a distributed and multi-tenant gateway to provide serverless SQL on Data
Warehouses and Lakehouses.
Apache Kyuubi is providing varied protocols like Thrift, Trino and etc., to the engines as Spark, Flink, Hive, JDBC and
etc.
Drois is supported as JDBC data source in Apache Kyuubi.
Apache Kyuubi also provides a serious useful feature with HA, service discovry,
unified authentication, engine lifecycles and etc.

## Usage

### Download Apache Kyuubi

Download Apache Kyuubi from <https://kyuubi.apache.org/zh/releases.html>

Get Apache Kyuubi 1.6.0 or above and extract it to folder。

### Config Doris as Kyuubi data source

- Update Kyuubi configurations in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`

```properties
kyuubi.engine.type=jdbc
kyuubi.engine.jdbc.type=doris
kyuubi.engine.jdbc.driver.class=com.mysql.cj.jdbc.Driver
kyuubi.engine.jdbc.connection.url=jdbc:mysql://xxx:xxx
kyuubi.engine.jdbc.connection.user=***
kyuubi.engine.jdbc.connection.password=***
```

| Configuration | Description |
|----------------------------------------|---------------------------------------------------------------|
| kyuubi.engine.type | Enine Type, specify to `jdbc` |
| kyuubi.engine.jdbc.type | JDBC service type, specify to `doris` |
| kyuubi.engine.jdbc.connection.url | JDBC url to Doris FE |
| kyuubi.engine.jdbc.connection.user | JDBC username |
| kyuubi.engine.jdbc.connection.password | JDBC password |
| kyuubi.engine.jdbc.driver.class | JDBC driver class name, specify to `com.mysql.cj.jdbc.Driver` |

- For other configuration in Apache Kyuubi, please refer
to [Apache Kyuubi Configuration Docs](https://kyuubi.readthedocs.io/en/master/deployment/settings.html)

### Add MySQL JDBC Driver

Copy the Mysql JDBC Driver `mysql-connector-j-8.X.X.jar` to `$KYUUBI_HOME/incubating-bin/externals/engines/jdbc` 目录下。

### Start Kyuubi Server

Run `$KYUUBI_HOME/bin/kyuubi run`.
After started, port 10009 by default is listened by Kyuubi Server with Thrift protocol。

## Example

The fowllowing example shows basic example of querying Doris with Kyuubi with beeline CLI in thift protocol.

### Connect to Kyuubi with Beeline

```shell
$ ./beeline -u "jdbc:hive2://xxxx:10009/"
```

### Execute Query to Kyuubi

Execute query statement `select * from demo.expamle_tbl;` with query results returned.

```shell
0: jdbc:hive2://xxxx:10009/> select * from demo.example_tbl;

2023-03-07 09:29:14.771 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[bdc59dd0-ceea-4c02-8c3a-23424323f5db]: PENDING_STATE -> RUNNING_STATE, statement:
select * from demo.example_tbl
2023-03-07 09:29:14.786 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[bdc59dd0-ceea-4c02-8c3a-23424323f5db] in FINISHED_STATE
2023-03-07 09:29:14.787 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[bdc59dd0-ceea-4c02-8c3a-23424323f5db]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.015 seconds
+----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+
| user_id | date | city | age | sex | last_visit_date | cost | max_dwell_time | min_dwell_time |
+----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+
| 10000 | 2017-10-01 | 北京 | 20 | 0 | 2017-10-01 07:00:00.0 | 70 | 10 | 2 |
| 10001 | 2017-10-01 | 北京 | 30 | 1 | 2017-10-01 17:05:45.0 | 4 | 22 | 22 |
| 10002 | 2017-10-02 | 上海 | 20 | 1 | 2017-10-02 12:59:12.0 | 400 | 5 | 5 |
| 10003 | 2017-10-02 | 广州 | 32 | 0 | 2017-10-02 11:20:00.0 | 60 | 11 | 11 |
| 10004 | 2017-10-01 | 深圳 | 35 | 0 | 2017-10-01 10:00:15.0 | 200 | 3 | 3 |
| 10004 | 2017-10-03 | 深圳 | 35 | 0 | 2017-10-03 10:20:22.0 | 22 | 6 | 6 |
+----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+
6 rows selected (0.068 seconds)
```
1 change: 1 addition & 0 deletions docs/sidebars.json
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,7 @@
"ecosystem/spark-doris-connector",
"ecosystem/flink-doris-connector",
"ecosystem/datax",
"ecosystem/kyuubi",
"ecosystem/mysql-to-doris",
"ecosystem/logstash",
"ecosystem/plugin-development-manual",
Expand Down
114 changes: 114 additions & 0 deletions docs/zh-CN/docs/ecosystem/kyuubi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
---

{
"title": "Kyuubi",
"language": "zh-CN"
}

---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# 通过 Kyuubi 连接 Doris

## 介绍

[Apache Kyuubi](https://kyuubi.apache.org/),一个分布式和多租户网关,用于在 Lakehouse 上提供 Serverless
SQL,可连接包括Spark、Flink、Hive、JDBC等引擎,并对外提供Thrift、Trino等接口协议供灵活对接。
其中Apache Kyuubi实现了JDBC Engine并支持Doris方言,并可用于对接Doris作为数据源。
Apache Kyuubi可提供高可用、服务发现、租户隔离、统一认证、生命周期等一系列特性。

## 下载 Apache Kyuubi

## 配置方法

### 下载 Apache Kyuubi

从官网下载Apache Kyuubi 1.6.0或以上版本的安装包后解压。

Apache Kyuubi 下载地址: <https://kyuubi.apache.org/zh/releases.html>

### 配置Doris作为Kyuubi数据源

- 修改配置文件 `$KYUUBI_HOME/conf/kyuubi-defaults.conf`

```properties
kyuubi.engine.type=jdbc
kyuubi.engine.jdbc.type=doris
kyuubi.engine.jdbc.driver.class=com.mysql.cj.jdbc.Driver
kyuubi.engine.jdbc.connection.url=jdbc:mysql://xxx:xxx
kyuubi.engine.jdbc.connection.user=***
kyuubi.engine.jdbc.connection.password=***
```

| 配置项 | 说明 |
|----------------------------------------|-----------------------------------------------|
| kyuubi.engine.type | 引擎类型。请使用jdbc |
| kyuubi.engine.jdbc.type | jdbc服务类型。这里请指定为doris |
| kyuubi.engine.jdbc.connection.url | jdbc 服务连接。这里请指定 Doris FE 上的 mysql server 连接地址 |
| kyuubi.engine.jdbc.connection.user | jdbc 服务用户名 |
| kyuubi.engine.jdbc.connection.password | jdbc 服务密码 |
| kyuubi.engine.jdbc.driver.class | 连接jdbc服务使用的驱动类名。请使用com.mysql.cj.jdbc.Driver |

- 其他相关配置参考 [Apache Kyuubi配置说明](https://kyuubi.readthedocs.io/en/master/deployment/settings.html)

### 添加MySQL驱动

添加 Mysql JDB C驱动 `mysql-connector-j-8.X.X.jar` 到 `$KYUUBI_HOME/incubating-bin/externals/engines/jdbc` 目录下。

### 启动 Kyuubi 服务

`$KYUUBI_HOME/bin/kyuubi run`
启动后,Kyuubi默认监听10009端口提供Thrift协议。

## 使用方法

以下例子展示通过Apache Kyuubi的beeline工具经Thrift协议查询Doris。

### 建立连接

```shell
$ ./beeline -u "jdbc:hive2://xxxx:10009/"

```

### 执行查询

执行查询语句 `select * from demo.expamle_tbl;` 并得到结果。

```shell
0: jdbc:hive2://xxxx:10009/> select * from demo.example_tbl;

2023-03-07 09:29:14.771 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[bdc59dd0-ceea-4c02-8c3a-23424323f5db]: PENDING_STATE -> RUNNING_STATE, statement:
select * from demo.example_tbl
2023-03-07 09:29:14.786 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[bdc59dd0-ceea-4c02-8c3a-23424323f5db] in FINISHED_STATE
2023-03-07 09:29:14.787 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[bdc59dd0-ceea-4c02-8c3a-23424323f5db]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.015 seconds
+----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+
| user_id | date | city | age | sex | last_visit_date | cost | max_dwell_time | min_dwell_time |
+----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+
| 10000 | 2017-10-01 | 北京 | 20 | 0 | 2017-10-01 07:00:00.0 | 70 | 10 | 2 |
| 10001 | 2017-10-01 | 北京 | 30 | 1 | 2017-10-01 17:05:45.0 | 4 | 22 | 22 |
| 10002 | 2017-10-02 | 上海 | 20 | 1 | 2017-10-02 12:59:12.0 | 400 | 5 | 5 |
| 10003 | 2017-10-02 | 广州 | 32 | 0 | 2017-10-02 11:20:00.0 | 60 | 11 | 11 |
| 10004 | 2017-10-01 | 深圳 | 35 | 0 | 2017-10-01 10:00:15.0 | 200 | 3 | 3 |
| 10004 | 2017-10-03 | 深圳 | 35 | 0 | 2017-10-03 10:20:22.0 | 22 | 6 | 6 |
+----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+
6 rows selected (0.068 seconds)
```