Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/.vuepress/sidebar/en.js
Original file line number Diff line number Diff line change
Expand Up @@ -579,6 +579,8 @@ module.exports = [
"DROP VIEW",
"HLL",
"RECOVER",
"REFRESH DATABASE",
"REFRESH TABLE",
"RESTORE",
"SHOW ENCRYPTKEYS",
"TRUNCATE TABLE",
Expand Down
2 changes: 2 additions & 0 deletions docs/.vuepress/sidebar/zh-CN.js
Original file line number Diff line number Diff line change
Expand Up @@ -584,6 +584,8 @@ module.exports = [
"DROP VIEW",
"HLL",
"RECOVER",
"REFRESH DATABASE",
"REFRESH TABLE",
"RESTORE",
"SHOW ENCRYPTKEYS",
"SHOW RESOURCES",
Expand Down
67 changes: 66 additions & 1 deletion docs/en/extending-doris/iceberg-of-doris.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,8 +97,42 @@ Iceberg tables can be created in Doris in two ways. You do not need to declare t

The progress of the table build in `iceberg_test_db` can be viewed by `HELP SHOW TABLE CREATION`.


You can also create an Iceberg table by explicitly specifying the column definitions according to your needs.

1. Create an Iceberg table

```sql
-- Syntax
CREATE [EXTERNAL] TABLE table_name (
col_name col_type [NULL | NOT NULL] [COMMENT "comment"]
) ENGINE = ICEBERG
[COMMENT "comment"] )
PROPERTIES (
"iceberg.database" = "iceberg_db_name",
"iceberg.table" = "icberg_table_name",
"iceberg.hive.metastore.uris" = "thrift://192.168.0.1:9083",
"iceberg.catalog.type" = "HIVE_CATALOG"
);

-- Example: Mount iceberg_table under iceberg_db in Iceberg
CREATE TABLE `t_iceberg` (
`id` int NOT NULL COMMENT "id number",
`name` varchar(10) NOT NULL COMMENT "user name"
) ENGINE = ICEBERG
PROPERTIES (
"iceberg.database" = "iceberg_db",
"iceberg.table" = "iceberg_table",
"iceberg.hive.metastore.uris" = "thrift://192.168.0.1:9083",
"iceberg.catalog.type" = "HIVE_CATALOG"
);
```

#### Parameter Description

- External Table Columns
- Column names should correspond to the Iceberg table
- The order of the columns needs to be consistent with the Iceberg table
- ENGINE needs to be specified as ICEBERG
- PROPERTIES property.
- `iceberg.hive.metastore.uris`: Hive Metastore service address
Expand All @@ -110,6 +144,18 @@ Iceberg tables can be created in Doris in two ways. You do not need to declare t

Show table structure can be viewed by `HELP SHOW CREATE TABLE`.

### Synchronized mounts

When the Iceberg table Schema changes, you can manually synchronize it with the `REFRESH` command, which will remove and rebuild the Iceberg external table in Doris, as seen in the `HELP REFRESH` help.

```sql
-- Synchronize the Iceberg table
REFRESH TABLE t_iceberg;

-- Synchronize the Iceberg database
REFRESH DATABASE iceberg_test_db;
```

## Data Type Matching

The supported Iceberg column types correspond to Doris in the following table.
Expand All @@ -134,7 +180,7 @@ The supported Iceberg column types correspond to Doris in the following table.
| MAP | - | not supported |

**Note:**
- Iceberg table Schema changes **are not automatically synchronized** and require rebuilding the Iceberg external tables or database in Doris.
- Iceberg table Schema changes **are not automatically synchronized** and require synchronization of Iceberg external tables or databases in Doris via the `REFRESH` command.
- The current default supported version of Iceberg is 0.12.0 and has not been tested in other versions. More versions will be supported in the future.

### Query Usage
Expand All @@ -144,3 +190,22 @@ Once you have finished building the Iceberg external table in Doris, it is no di
```sql
select * from t_iceberg where k1 > 1000 and k3 = 'term' or k4 like '%doris';
```

## Related system configurations

### FE Configuration

The following configurations are at the Iceberg external table system level and can be configured by modifying `fe.conf` or by `ADMIN SET CONFIG`.

- `iceberg_table_creation_strict_mode`

Iceberg tables are created with strict mode enabled by default.
strict mode means that the column types of the Iceberg table are strictly filtered, and if there are data types that Doris does not currently support, the creation of the table will fail.

- `iceberg_table_creation_interval_second`

The background task execution interval for automatic creation of Iceberg tables, default is 10s.

- `max_iceberg_table_creation_record_size`

The maximum value reserved for Iceberg table creation records, default is 2000. Only for creating Iceberg database records.
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
{
"title": "REFRESH DATABASE",
"language": "en"
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# REFRESH DATABASE

## Description

This statement is used to synchronize the remote Iceberg database and will delete and rebuild the Iceberg tables under the current Doris database, leaving the non-Iceberg tables unaffected.
Syntax:
REFRESH DATABASE db_name;

Instructions.
1) Valid only for the Iceberg database mounted in Doris.

## Example

1) Refresh the database iceberg_test_db
REFRESH DATABASE iceberg_test_db;

## keyword

REFRESH,DATABASE
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
{
"title": "REFRESH TABLE",
"language": "en"
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# REFRESH TABLE

## Description

This statement is used to synchronize a remote Iceberg table and will delete and rebuild Doris' current external table.
Syntax.
REFRESH TABLE tbl_name;

Instructions.
1) Valid only for the Iceberg table mounted in Doris.

## Example

1) Refresh the table iceberg_tbl
REFRESH TABLE iceberg_tbl;

## keyword

REFRESH,TABLE
66 changes: 65 additions & 1 deletion docs/zh-CN/extending-doris/iceberg-of-doris.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,8 +97,41 @@ Iceberg External Table of Doris 提供了 Doris 直接访问 Iceberg 外部表

`iceberg_test_db` 中的建表进度可以通过 `HELP SHOW TABLE CREATION` 查看。

也可以根据自己的需求明确指定列定义来创建 Iceberg 外表。

1. 创一个 Iceberg 外表

```sql
-- 语法
CREATE [EXTERNAL] TABLE table_name (
col_name col_type [NULL | NOT NULL] [COMMENT "comment"]
) ENGINE = ICEBERG
[COMMENT "comment"]
PROPERTIES (
"iceberg.database" = "iceberg_db_name",
"iceberg.table" = "icberg_table_name",
"iceberg.hive.metastore.uris" = "thrift://192.168.0.1:9083",
"iceberg.catalog.type" = "HIVE_CATALOG"
);

-- 例子:挂载 Iceberg 中 iceberg_db 下的 iceberg_table
CREATE TABLE `t_iceberg` (
`id` int NOT NULL COMMENT "id number",
`name` varchar(10) NOT NULL COMMENT "user name"
) ENGINE = ICEBERG
PROPERTIES (
"iceberg.database" = "iceberg_db",
"iceberg.table" = "iceberg_table",
"iceberg.hive.metastore.uris" = "thrift://192.168.0.1:9083",
"iceberg.catalog.type" = "HIVE_CATALOG"
);
```

#### 参数说明:

- 外表列
- 列名要于 Iceberg 表一一对应
- 列的顺序需要与 Iceberg 表一致
- ENGINE 需要指定为 ICEBERG
- PROPERTIES 属性:
- `iceberg.hive.metastore.uris`:Hive Metastore 服务地址
Expand All @@ -109,6 +142,18 @@ Iceberg External Table of Doris 提供了 Doris 直接访问 Iceberg 外部表
### 展示表结构

展示表结构可以通过 `HELP SHOW CREATE TABLE` 查看。

### 同步挂载

当 Iceberg 表 Schema 发生变更时,可以通过 `REFRESH` 命令手动同步,该命令会将 Doris 中的 Iceberg 外表删除重建,具体帮助可以通过 `HELP REFRESH` 查看。

```sql
-- 同步 Iceberg 表
REFRESH TABLE t_iceberg;

-- 同步 Iceberg 数据库
REFRESH DATABASE iceberg_test_db;
```

## 类型匹配

Expand All @@ -134,7 +179,7 @@ Iceberg External Table of Doris 提供了 Doris 直接访问 Iceberg 外部表
| MAP | - | 不支持 |

**注意:**
- Iceberg 表 Schema 变更**不会自动同步**,需要在 Doris 中重建 Iceberg 外表或数据库。
- Iceberg 表 Schema 变更**不会自动同步**,需要在 Doris 中通过 `REFRESH` 命令同步 Iceberg 外表或数据库。
- 当前默认支持的 Iceberg 版本为 0.12.0,未在其他版本进行测试。后续后支持更多版本。

### 查询用法
Expand All @@ -144,3 +189,22 @@ Iceberg External Table of Doris 提供了 Doris 直接访问 Iceberg 外部表
```sql
select * from t_iceberg where k1 > 1000 and k3 ='term' or k4 like '%doris';
```

## 相关系统配置

### FE配置

下面几个配置属于 Iceberg 外表系统级别的配置,可以通过修改 `fe.conf` 来配置,也可以通过 `ADMIN SET CONFIG` 来配置。

- `iceberg_table_creation_strict_mode`

创建 Iceberg 表默认开启 strict mode。
strict mode 是指对 Iceberg 表的列类型进行严格过滤,如果有 Doris 目前不支持的数据类型,则创建外表失败。

- `iceberg_table_creation_interval_second`

自动创建 Iceberg 表的后台任务执行间隔,默认为 10s。

- `max_iceberg_table_creation_record_size`

Iceberg 表创建记录保留的最大值,默认为 2000. 仅针对创建 Iceberg 数据库记录。
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
{
"title": "REFRESH DATABASE",
"language": "zh-CN"
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# REFRESH DATABASE

## Description

该语句用于同步远端 Iceberg 数据库,会将 Doris 当前数据库下的 Iceberg 外表删除重建,非 Iceberg 外表不受影响。
语法:
REFRESH DATABASE db_name;

说明:
1) 仅针对 Doris 中挂载的 Iceberg 数据库有效。

## Example

1. 刷新数据库 iceberg_test_db
REFRESH DATABASE iceberg_test_db;

## keyword

REFRESH,DATABASE

Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
{
"title": "REFRESH TABLE",
"language": "zh-CN"
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# REFRESH TABLE

## Description

该语句用于同步远端 Iceberg 表,会将 Doris 当前的外表删除重建。
语法:
REFRESH TABLE tbl_name;

说明:
1) 仅针对 Doris 中挂载的 Iceberg 表有效。

## Example

1. 刷新表 iceberg_tbl
REFRESH TABLE iceberg_tbl;

## keyword

REFRESH,TABLE

Loading