From a89bc0d2e6f500718d3ea646b629a98248701fa6 Mon Sep 17 00:00:00 2001 From: liangbowen Date: Tue, 7 Mar 2023 09:32:10 +0800 Subject: [PATCH 1/7] add kyuubi doc of zh-CN --- docs/zh-CN/docs/ecosystem/kyuubi.md | 110 ++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 docs/zh-CN/docs/ecosystem/kyuubi.md diff --git a/docs/zh-CN/docs/ecosystem/kyuubi.md b/docs/zh-CN/docs/ecosystem/kyuubi.md new file mode 100644 index 00000000000000..39e9471b96d4e4 --- /dev/null +++ b/docs/zh-CN/docs/ecosystem/kyuubi.md @@ -0,0 +1,110 @@ +--- + +{ +"title": "Kyuubi 连接 Doris", +"language": "zh-CN" +} + +--- + + + +# Kyuubi 连接 Doris + +## 介绍 + +[Apache Kyuubi](https://kyuubi.apache.org/)，一个分布式和多租户网关，用于在 Lakehouse 上提供 Serverless +SQL，可连接包括Spark、Flink、Hive、JDBC等引擎，并对外提供Thrift、Trino等接口协议供灵活对接。 +其中Apache Kyuubi实现了JDBC Engine并支持Doris方言，并可用于对接Doris作为数据源。 +Apache Kyuubi可提供高可用、服务发现、租户隔离、统一认证、生命周期等一系列特性。 + +## 下载 Apache Kyuubi + + + +## 配置方法 + +### 下载 Apache Kyuubi + +从官网下载Apache Kyuubi 1.6.0或以上版本的安装包后解压。 +Apache Kyuubi 下载地址： [Apache Kyuubi 最新版下载地址](https://kyuubi.apache.org/zh/releases.html) + +### 配置Doris作为Kyuubi数据源 + +- 修改配置文件 `$KYUUBI_HOME/conf/kyuubi-defaults.conf` + +```properties +kyuubi.engine.type=jdbc +kyuubi.engine.jdbc.type=doris +kyuubi.engine.jdbc.driver.class=com.mysql.cj.jdbc.Driver +kyuubi.engine.jdbc.connection.url=jdbc:mysql://xxx:xxx +kyuubi.engine.jdbc.connection.user=*** +kyuubi.engine.jdbc.connection.password=*** +``` + +| 配置项 | 说明 | +|----------------------------------------|-----------------------------------------------| +| kyuubi.engine.type | 引擎类型。请使用jdbc | +| kyuubi.engine.jdbc.type | jdbc服务类型。这里请指定为doris | +| kyuubi.engine.jdbc.connection.url | jdbc 服务连接。这里请指定 Doris FE 上的 mysql server 连接地址 | +| kyuubi.engine.jdbc.connection.user | jdbc 服务用户名 | +| kyuubi.engine.jdbc.connection.password | jdbc 服务密码 | +| kyuubi.engine.jdbc.driver.class | 连接jdbc服务使用的驱动类名。请使用com.mysql.cj.jdbc.Driver | + +- 其他相关配置参考 [Apache Kyuubi配置说明](https://kyuubi.readthedocs.io/en/master/deployment/settings.html) + +### 添加MySQL驱动 +添加Mysql JDBC驱动 `mysql-connector-java-8.X.X.jar` 到 `$KYUUBI_HOME/incubating-bin/externals/engines/jdbc` 目录下。 + +### 启动 Kyuubi 服务 +`$KYUUBI_HOME/bin/kyuubi run` +启动后，Kyuubi默认监听1009端口提供Thrift协议。 + +## 使用方法 + +以下例子展示通过Apache Kyuubi的beeline工具经Thrift协议查询Doris。 + +### 建立连接 +```shell +$ ./beeline -u "jdbc:hive2://xxxx:10009/;#kyuubi.engine.type=jdbc"` +``` + +### 执行查询 +执行查询语句 `select * from demo.expamle_tbl;` 并得到结果， + +```shell +0: jdbc:hive2://xxxx:10009/> select * from demo.example_tbl; + +2023-03-07 09:29:14.771 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[bdc59dd0-ceea-4c02-8c3a-23424323f5db]: PENDING_STATE -> RUNNING_STATE, statement: +select * from demo.example_tbl +2023-03-07 09:29:14.786 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[bdc59dd0-ceea-4c02-8c3a-23424323f5db] in FINISHED_STATE +2023-03-07 09:29:14.787 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[bdc59dd0-ceea-4c02-8c3a-23424323f5db]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.015 seconds ++----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+ +| user_id | date | city | age | sex | last_visit_date | cost | max_dwell_time | min_dwell_time | ++----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+ +| 10000 | 2017-10-01 | 北京 | 20 | 0 | 2017-10-01 07:00:00.0 | 70 | 10 | 2 | +| 10001 | 2017-10-01 | 北京 | 30 | 1 | 2017-10-01 17:05:45.0 | 4 | 22 | 22 | +| 10002 | 2017-10-02 | 上海 | 20 | 1 | 2017-10-02 12:59:12.0 | 400 | 5 | 5 | +| 10003 | 2017-10-02 | 广州 | 32 | 0 | 2017-10-02 11:20:00.0 | 60 | 11 | 11 | +| 10004 | 2017-10-01 | 深圳 | 35 | 0 | 2017-10-01 10:00:15.0 | 200 | 3 | 3 | +| 10004 | 2017-10-03 | 深圳 | 35 | 0 | 2017-10-03 10:20:22.0 | 22 | 6 | 6 | ++----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+ +6 rows selected (0.068 seconds) +``` From 15b8aa1c271c53b8a34b47ea85ffdf5a47c92a79 Mon Sep 17 00:00:00 2001 From: liangbowen Date: Tue, 7 Mar 2023 09:57:35 +0800 Subject: [PATCH 2/7] add kyuubi doc of en --- docs/en/docs/ecosystem/kyuubi.md | 111 ++++++++++++++++++++++++++++ docs/zh-CN/docs/ecosystem/kyuubi.md | 17 +++-- 2 files changed, 121 insertions(+), 7 deletions(-) create mode 100644 docs/en/docs/ecosystem/kyuubi.md diff --git a/docs/en/docs/ecosystem/kyuubi.md b/docs/en/docs/ecosystem/kyuubi.md new file mode 100644 index 00000000000000..288b1d2bd05278 --- /dev/null +++ b/docs/en/docs/ecosystem/kyuubi.md @@ -0,0 +1,111 @@ + +--- + +{ +"title": "Kyuubi", +"language": "zh-CN" +} + +--- + + + +# Use Kyuubi with Doris + +## Introduction + +[Apache Kyuubi](https://kyuubi.apache.org/) is a distributed and multi-tenant gateway to provide serverless SQL on Data Warehouses and Lakehouses. +Apache Kyuubi is providing varied protocols like Thrift, Trino and etc., to the engines as Spark, Flink, Hive, JDBC and etc. +Drois is supported as JDBC data source in Apache Kyuubi. +Apache Kyuubi also provides a serious useful feature with HA, service discovry, +unified authentication, engine lifecycles and etc. + + +## Usage + +### Download Apache Kyuubi + +Download Apache Kyuubi from + +Get Apache Kyuubi 1.6.0 or above and extract it to folder。 + + +### Config Doris as Kyuubi data source + +- Update Kyuubi configurations in `$KYUUBI_HOME/conf/kyuubi-defaults.conf` + +```properties +kyuubi.engine.type=jdbc +kyuubi.engine.jdbc.type=doris +kyuubi.engine.jdbc.driver.class=com.mysql.cj.jdbc.Driver +kyuubi.engine.jdbc.connection.url=jdbc:mysql://xxx:xxx +kyuubi.engine.jdbc.connection.user=*** +kyuubi.engine.jdbc.connection.password=*** +``` + +| Configuration | Description | +|----------------------------------------|-----------------------------------------------| +| kyuubi.engine.type | Enine Type, specify to `jdbc` | +| kyuubi.engine.jdbc.type | JDBC service type, specify to `doris` | +| kyuubi.engine.jdbc.connection.url | JDBC url to Doris FE | +| kyuubi.engine.jdbc.connection.user | JDBC username | +| kyuubi.engine.jdbc.connection.password | JDBC password | +| kyuubi.engine.jdbc.driver.class | JDBC driver class name, specify to `com.mysql.cj.jdbc.Driver` | + +- For other configuration in Apache Kyuubi, please refer to [Apache Kyuubi Configuration Docs](https://kyuubi.readthedocs.io/en/master/deployment/settings.html) + +### Add MySQL JDBC Driver +Copy the Mysql JDBC Driver `mysql-connector-j-8.X.X.jar` to `$KYUUBI_HOME/incubating-bin/externals/engines/jdbc` 目录下。 + +### Start Kyuubi Server +Run `$KYUUBI_HOME/bin/kyuubi run`. +After started, port 10009 by default is listened by Kyuubi Server with Thrift protocol。 + +## Example + +The fowllowing example shows basic example of querying Doris with Kyuubi with beeline CLI in thift protocol. + +### Connect to Kyuubi with Beeline +```shell +$ ./beeline -u "jdbc:hive2://xxxx:10009/;#kyuubi.engine.type=jdbc"` +``` + +### Execute Query to Kyuubi +Execute query statement `select * from demo.expamle_tbl;` with query results returned. + +```shell +0: jdbc:hive2://xxxx:10009/> select * from demo.example_tbl; + +2023-03-07 09:29:14.771 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[bdc59dd0-ceea-4c02-8c3a-23424323f5db]: PENDING_STATE -> RUNNING_STATE, statement: +select * from demo.example_tbl +2023-03-07 09:29:14.786 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[bdc59dd0-ceea-4c02-8c3a-23424323f5db] in FINISHED_STATE +2023-03-07 09:29:14.787 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[bdc59dd0-ceea-4c02-8c3a-23424323f5db]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.015 seconds ++----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+ +| user_id | date | city | age | sex | last_visit_date | cost | max_dwell_time | min_dwell_time | ++----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+ +| 10000 | 2017-10-01 | 北京 | 20 | 0 | 2017-10-01 07:00:00.0 | 70 | 10 | 2 | +| 10001 | 2017-10-01 | 北京 | 30 | 1 | 2017-10-01 17:05:45.0 | 4 | 22 | 22 | +| 10002 | 2017-10-02 | 上海 | 20 | 1 | 2017-10-02 12:59:12.0 | 400 | 5 | 5 | +| 10003 | 2017-10-02 | 广州 | 32 | 0 | 2017-10-02 11:20:00.0 | 60 | 11 | 11 | +| 10004 | 2017-10-01 | 深圳 | 35 | 0 | 2017-10-01 10:00:15.0 | 200 | 3 | 3 | +| 10004 | 2017-10-03 | 深圳 | 35 | 0 | 2017-10-03 10:20:22.0 | 22 | 6 | 6 | ++----------+-------------+-------+------+------+------------------------+-------+-----------------+-----------------+ +6 rows selected (0.068 seconds) +``` diff --git a/docs/zh-CN/docs/ecosystem/kyuubi.md b/docs/zh-CN/docs/ecosystem/kyuubi.md index 39e9471b96d4e4..541e57ad7dc753 100644 --- a/docs/zh-CN/docs/ecosystem/kyuubi.md +++ b/docs/zh-CN/docs/ecosystem/kyuubi.md @@ -1,7 +1,7 @@ --- { -"title": "Kyuubi 连接 Doris", +"title": "Kyuubi", "language": "zh-CN" } @@ -26,7 +26,7 @@ specific language governing permissions and limitations under the License. --> -# Kyuubi 连接 Doris +# 通过 Kyuubi 连接 Doris ## 介绍 @@ -37,14 +37,13 @@ Apache Kyuubi可提供高可用、服务发现、租户隔离、统一认证、 ## 下载 Apache Kyuubi - - ## 配置方法 ### 下载 Apache Kyuubi 从官网下载Apache Kyuubi 1.6.0或以上版本的安装包后解压。 -Apache Kyuubi 下载地址： [Apache Kyuubi 最新版下载地址](https://kyuubi.apache.org/zh/releases.html) + +Apache Kyuubi 下载地址： ### 配置Doris作为Kyuubi数据源 @@ -71,22 +70,26 @@ kyuubi.engine.jdbc.connection.password=*** - 其他相关配置参考 [Apache Kyuubi配置说明](https://kyuubi.readthedocs.io/en/master/deployment/settings.html) ### 添加MySQL驱动 -添加Mysql JDBC驱动 `mysql-connector-java-8.X.X.jar` 到 `$KYUUBI_HOME/incubating-bin/externals/engines/jdbc` 目录下。 + +添加 Mysql JDB C驱动 `mysql-connector-j-8.X.X.jar` 到 `$KYUUBI_HOME/incubating-bin/externals/engines/jdbc` 目录下。 ### 启动 Kyuubi 服务 + `$KYUUBI_HOME/bin/kyuubi run` -启动后，Kyuubi默认监听1009端口提供Thrift协议。 +启动后，Kyuubi默认监听10009端口提供Thrift协议。 ## 使用方法 以下例子展示通过Apache Kyuubi的beeline工具经Thrift协议查询Doris。 ### 建立连接 + ```shell $ ./beeline -u "jdbc:hive2://xxxx:10009/;#kyuubi.engine.type=jdbc"` ``` ### 执行查询 + 执行查询语句 `select * from demo.expamle_tbl;` 并得到结果， ```shell From 76eeb269632ce4e414c7cfa5edaeda0578691600 Mon Sep 17 00:00:00 2001 From: liangbowen Date: Tue, 7 Mar 2023 10:04:06 +0800 Subject: [PATCH 3/7] update sidebars --- docs/sidebars.json | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/sidebars.json b/docs/sidebars.json index 363a7da352915c..dcc78f198212a6 100644 --- a/docs/sidebars.json +++ b/docs/sidebars.json @@ -224,6 +224,7 @@ "ecosystem/spark-doris-connector", "ecosystem/flink-doris-connector", "ecosystem/datax", + "ecosystem/kyuubi", "ecosystem/mysql-to-doris", "ecosystem/logstash", "ecosystem/plugin-development-manual", From 0ce8722eb4b3ef23fc0228846473b4f31d2ffa37 Mon Sep 17 00:00:00 2001 From: liangbowen Date: Tue, 7 Mar 2023 10:21:24 +0800 Subject: [PATCH 4/7] fix language typo --- docs/en/docs/ecosystem/kyuubi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/docs/ecosystem/kyuubi.md b/docs/en/docs/ecosystem/kyuubi.md index 288b1d2bd05278..5157e72d6e4510 100644 --- a/docs/en/docs/ecosystem/kyuubi.md +++ b/docs/en/docs/ecosystem/kyuubi.md @@ -3,7 +3,7 @@ { "title": "Kyuubi", -"language": "zh-CN" +"language": "en" } --- From e8eee551973820598be7035c5851f2ceef1d6b49 Mon Sep 17 00:00:00 2001 From: liangbowen Date: Tue, 7 Mar 2023 10:23:14 +0800 Subject: [PATCH 5/7] typo --- docs/zh-CN/docs/ecosystem/kyuubi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh-CN/docs/ecosystem/kyuubi.md b/docs/zh-CN/docs/ecosystem/kyuubi.md index 541e57ad7dc753..3d05133a528506 100644 --- a/docs/zh-CN/docs/ecosystem/kyuubi.md +++ b/docs/zh-CN/docs/ecosystem/kyuubi.md @@ -90,7 +90,7 @@ $ ./beeline -u "jdbc:hive2://xxxx:10009/;#kyuubi.engine.type=jdbc"` ### 执行查询 -执行查询语句 `select * from demo.expamle_tbl;` 并得到结果， +执行查询语句 `select * from demo.expamle_tbl;` 并得到结果。 ```shell 0: jdbc:hive2://xxxx:10009/> select * from demo.example_tbl; From 20246867645769c4796029f8fd858119850619a3 Mon Sep 17 00:00:00 2001 From: liangbowen Date: Tue, 7 Mar 2023 10:28:39 +0800 Subject: [PATCH 6/7] skip param in jdbc url --- docs/en/docs/ecosystem/kyuubi.md | 2 +- docs/zh-CN/docs/ecosystem/kyuubi.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/docs/ecosystem/kyuubi.md b/docs/en/docs/ecosystem/kyuubi.md index 5157e72d6e4510..4e466b575701bd 100644 --- a/docs/en/docs/ecosystem/kyuubi.md +++ b/docs/en/docs/ecosystem/kyuubi.md @@ -84,7 +84,7 @@ The fowllowing example shows basic example of querying Doris with Kyuubi with be ### Connect to Kyuubi with Beeline ```shell -$ ./beeline -u "jdbc:hive2://xxxx:10009/;#kyuubi.engine.type=jdbc"` +$ ./beeline -u "jdbc:hive2://xxxx:10009/" ``` ### Execute Query to Kyuubi diff --git a/docs/zh-CN/docs/ecosystem/kyuubi.md b/docs/zh-CN/docs/ecosystem/kyuubi.md index 3d05133a528506..861ed7ed1c647f 100644 --- a/docs/zh-CN/docs/ecosystem/kyuubi.md +++ b/docs/zh-CN/docs/ecosystem/kyuubi.md @@ -85,7 +85,7 @@ kyuubi.engine.jdbc.connection.password=*** ### 建立连接 ```shell -$ ./beeline -u "jdbc:hive2://xxxx:10009/;#kyuubi.engine.type=jdbc"` +$ ./beeline -u "jdbc:hive2://xxxx:10009/" ``` ### 执行查询 From 4a9302cec7ae71c57e67e1dd3e4391b14aea0fb6 Mon Sep 17 00:00:00 2001 From: liangbowen Date: Tue, 7 Mar 2023 10:37:16 +0800 Subject: [PATCH 7/7] style --- docs/en/docs/ecosystem/kyuubi.md | 32 ++++++++++++++++------------- docs/zh-CN/docs/ecosystem/kyuubi.md | 1 + 2 files changed, 19 insertions(+), 14 deletions(-) diff --git a/docs/en/docs/ecosystem/kyuubi.md b/docs/en/docs/ecosystem/kyuubi.md index 4e466b575701bd..618257874ad942 100644 --- a/docs/en/docs/ecosystem/kyuubi.md +++ b/docs/en/docs/ecosystem/kyuubi.md @@ -1,4 +1,3 @@ - --- { @@ -31,13 +30,14 @@ under the License. ## Introduction -[Apache Kyuubi](https://kyuubi.apache.org/) is a distributed and multi-tenant gateway to provide serverless SQL on Data Warehouses and Lakehouses. -Apache Kyuubi is providing varied protocols like Thrift, Trino and etc., to the engines as Spark, Flink, Hive, JDBC and etc. +[Apache Kyuubi](https://kyuubi.apache.org/) is a distributed and multi-tenant gateway to provide serverless SQL on Data +Warehouses and Lakehouses. +Apache Kyuubi is providing varied protocols like Thrift, Trino and etc., to the engines as Spark, Flink, Hive, JDBC and +etc. Drois is supported as JDBC data source in Apache Kyuubi. Apache Kyuubi also provides a serious useful feature with HA, service discovry, unified authentication, engine lifecycles and etc. - ## Usage ### Download Apache Kyuubi @@ -46,7 +46,6 @@ Download Apache Kyuubi from Get Apache Kyuubi 1.6.0 or above and extract it to folder。 - ### Config Doris as Kyuubi data source - Update Kyuubi configurations in `$KYUUBI_HOME/conf/kyuubi-defaults.conf` @@ -60,21 +59,24 @@ kyuubi.engine.jdbc.connection.user=*** kyuubi.engine.jdbc.connection.password=*** ``` -| Configuration | Description | -|----------------------------------------|-----------------------------------------------| -| kyuubi.engine.type | Enine Type, specify to `jdbc` | -| kyuubi.engine.jdbc.type | JDBC service type, specify to `doris` | -| kyuubi.engine.jdbc.connection.url | JDBC url to Doris FE | -| kyuubi.engine.jdbc.connection.user | JDBC username | -| kyuubi.engine.jdbc.connection.password | JDBC password | -| kyuubi.engine.jdbc.driver.class | JDBC driver class name, specify to `com.mysql.cj.jdbc.Driver` | +| Configuration | Description | +|----------------------------------------|---------------------------------------------------------------| +| kyuubi.engine.type | Enine Type, specify to `jdbc` | +| kyuubi.engine.jdbc.type | JDBC service type, specify to `doris` | +| kyuubi.engine.jdbc.connection.url | JDBC url to Doris FE | +| kyuubi.engine.jdbc.connection.user | JDBC username | +| kyuubi.engine.jdbc.connection.password | JDBC password | +| kyuubi.engine.jdbc.driver.class | JDBC driver class name, specify to `com.mysql.cj.jdbc.Driver` | -- For other configuration in Apache Kyuubi, please refer to [Apache Kyuubi Configuration Docs](https://kyuubi.readthedocs.io/en/master/deployment/settings.html) +- For other configuration in Apache Kyuubi, please refer + to [Apache Kyuubi Configuration Docs](https://kyuubi.readthedocs.io/en/master/deployment/settings.html) ### Add MySQL JDBC Driver + Copy the Mysql JDBC Driver `mysql-connector-j-8.X.X.jar` to `$KYUUBI_HOME/incubating-bin/externals/engines/jdbc` 目录下。 ### Start Kyuubi Server + Run `$KYUUBI_HOME/bin/kyuubi run`. After started, port 10009 by default is listened by Kyuubi Server with Thrift protocol。 @@ -83,11 +85,13 @@ After started, port 10009 by default is listened by Kyuubi Server with Thrift pr The fowllowing example shows basic example of querying Doris with Kyuubi with beeline CLI in thift protocol. ### Connect to Kyuubi with Beeline + ```shell $ ./beeline -u "jdbc:hive2://xxxx:10009/" ``` ### Execute Query to Kyuubi + Execute query statement `select * from demo.expamle_tbl;` with query results returned. ```shell diff --git a/docs/zh-CN/docs/ecosystem/kyuubi.md b/docs/zh-CN/docs/ecosystem/kyuubi.md index 861ed7ed1c647f..4e00f04f2235d8 100644 --- a/docs/zh-CN/docs/ecosystem/kyuubi.md +++ b/docs/zh-CN/docs/ecosystem/kyuubi.md @@ -86,6 +86,7 @@ kyuubi.engine.jdbc.connection.password=*** ```shell $ ./beeline -u "jdbc:hive2://xxxx:10009/" + ``` ### 执行查询