Skip to content

[feature](cold-hot) support remote storage #8532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
qidaye wants to merge 6 commits into from
Closed

[feature](cold-hot) support remote storage #8532

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e257a1d
support remote storage
Mar 15, 2022
394fec5
add modify remote storage clause and fix reviews
Mar 22, 2022
2a33c22
add alter tests, support alter single data property
Mar 28, 2022
743e6fa
fix imports and optimize if conditions
Mar 28, 2022
2d5f2dd
optimize if conditions
Mar 28, 2022
a14ca7f
optimize if conditions
Mar 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/.vuepress/sidebar/en.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -610,6 +610,7 @@ module.exports = [
"SHOW INDEX",
"SHOW MIGRATIONS",
"SHOW PLUGINS",
"SHOW REMOTE STORAGES",
"SHOW TABLE STATUS",
"SHOW TRASH",
"UNINSTALL PLUGIN",
Expand Down
1 change: 1 addition & 0 deletions docs/.vuepress/sidebar/zh-CN.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -623,6 +623,7 @@ module.exports = [
"SHOW INDEX",
"SHOW MIGRATIONS",
"SHOW PLUGINS",
"SHOW REMOTE STORAGES",
"SHOW TABLE STATUS",
"SHOW TRASH",
"UNINSTALL PLUGIN",
Expand Down
259 changes: 150 additions & 109 deletions docs/en/sql-reference/sql-statements/Administration/ALTER SYSTEM.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,112 +27,153 @@ under the License.
# ALTER SYSTEM
## Description

This statement is used to operate on nodes in a system. (Administrator only!)
Grammar:
1) Adding nodes (without multi-tenant functionality, add in this way)
ALTER SYSTEM ADD BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
2) Adding idle nodes (that is, adding BACKEND that does not belong to any cluster)
ALTER SYSTEM ADD FREE BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
3) Adding nodes to a cluster
ALTER SYSTEM ADD BACKEND TO cluster_name "host:heartbeat_port"[,"host:heartbeat_port"...];
4) Delete nodes
ALTER SYSTEM DROP BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
5) Node offline
ALTER SYSTEM DECOMMISSION BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
6)226;- 21152;-Broker
ALTER SYSTEM ADD BROKER broker_name "host:port"[,"host:port"...];
(7) 20943;"23569;" Broker
ALTER SYSTEM DROP BROKER broker_name "host:port"[,"host:port"...];
8) Delete all Brokers
ALTER SYSTEM DROP ALL BROKER broker_name
9) Set up a Load error hub for centralized display of import error information
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES ("key" = "value"[, ...]);
10) Modify property of BE
ALTER SYSTEM MODIFY BACKEND "host:heartbeat_port" SET ("key" = "value"[, ...]);

Explain:
1) Host can be hostname or IP address
2) heartbeat_port is the heartbeat port of the node
3) Adding and deleting nodes are synchronous operations. These two operations do not take into account the existing data on the node, the node is directly deleted from the metadata, please use cautiously.
4) Node offline operations are used to secure offline nodes. This operation is asynchronous. If successful, the node will eventually be removed from the metadata. If it fails, the offline will not be completed.
5) The offline operation of the node can be cancelled manually. See CANCEL DECOMMISSION for details
6) Load error hub:
Currently, two types of Hub are supported: Mysql and Broker. You need to specify "type" = "mysql" or "type" = "broker" in PROPERTIES.
If you need to delete the current load error hub, you can set type to null.
1) When using the Mysql type, the error information generated when importing will be inserted into the specified MySQL library table, and then the error information can be viewed directly through the show load warnings statement.

Hub of Mysql type needs to specify the following parameters:
host: mysql host
port: mysql port
user: mysql user
password: mysql password
database mysql database
table: mysql table

2) When the Broker type is used, the error information generated when importing will form a file and be written to the designated remote storage system through the broker. Make sure that the corresponding broker is deployed
Hub of Broker type needs to specify the following parameters:
Broker: Name of broker
Path: Remote Storage Path
Other properties: Other information necessary to access remote storage, such as authentication information.

7) Modify BE node attributes currently supports the following attributes:
1. tag.location：Resource tag
2. disable_query: Query disabled attribute
3. disable_load: Load disabled attribute

## example

1. Add a node
ALTER SYSTEM ADD BACKEND "host:port";

2. Adding an idle node
ALTER SYSTEM ADD FREE BACKEND "host:port";

3. Delete two nodes
ALTER SYSTEM DROP BACKEND "host1:port", "host2:port";

4. offline two nodes
ALTER SYSTEM DECOMMISSION BACKEND "host1:port", "host2:port";

5. Add two Hdfs Broker
ALTER SYSTEM ADD BROKER hdfs "host1:port", "host2:port";

6. Add a load error hub of Mysql type
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "mysql",
"host" = "192.168.1.17"
"port" = "3306",
"User" = "my" name,
"password" = "my_passwd",
"database" = "doris_load",
"table" = "load_errors"
);

7. 添加一个 Broker 类型的 load error hub
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "broker",
"Name" = BOS,
"path" = "bos://backup-cmy/logs",
"bos_endpoint" ="http://gz.bcebos.com",
"bos_accesskey" = "069fc278xxxxxx24ddb522",
"bos_secret_accesskey"="700adb0c6xxxxxx74d59eaa980a"
);

8. Delete the current load error hub
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "null");

9. Modify BE resource tag

ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("tag.location" = "group_a");

10. Modify the query disabled attribute of BE

ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("disable_query" = "true");

11. Modify the load disabled attribute of BE

ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("disable_load" = "true");

## keyword
AGE,SYSTEM,BACKGROUND,BROKER,FREE
This statement is used to operate on nodes in a system. (Administrator only!)
Grammar:
1) Adding nodes (without multi-tenant functionality, add in this way)
ALTER SYSTEM ADD BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
2) Adding idle nodes (that is, adding BACKEND that does not belong to any cluster)
ALTER SYSTEM ADD FREE BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
3) Adding nodes to a cluster
ALTER SYSTEM ADD BACKEND TO cluster_name "host:heartbeat_port"[,"host:heartbeat_port"...];
4) Delete nodes
ALTER SYSTEM DROP BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
5) Node offline
ALTER SYSTEM DECOMMISSION BACKEND "host:heartbeat_port"[,"host:heartbeat_port"...];
6) Add Broker
ALTER SYSTEM ADD BROKER broker_name "host:port"[,"host:port"...];
7) Drop Broker
ALTER SYSTEM DROP BROKER broker_name "host:port"[,"host:port"...];
8) Delete all Brokers
ALTER SYSTEM DROP ALL BROKER broker_name
9) Set up a Load error hub for centralized display of import error information
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES ("key" = "value"[, ...]);
10) Modify property of BE
ALTER SYSTEM MODIFY BACKEND "host:heartbeat_port" SET ("key" = "value"[, ...]);
11) Add remote storage
ALTER SYSTEM ADD REMOTE STORAGE storage_name PROPERTIES ("key" = "value"[, ...]);
12) Drop remote storage
ALTER SYSTEM DROP REMOTE STORAGE storage_name;
13) Modify remote storage
ALTER SYSTEM MODIFY REMOTE STORAGE storage_name PROPERTIES ("key" = "value"[, ...]);

Explain:
1) Host can be hostname or IP address
2) heartbeat_port is the heartbeat port of the node
3) Adding and deleting nodes are synchronous operations. These two operations do not take into account the existing data on the node, the node is directly deleted from the metadata, please use cautiously.
4) Node offline operations are used to secure offline nodes. This operation is asynchronous. If successful, the node will eventually be removed from the metadata. If it fails, the offline will not be completed.
5) The offline operation of the node can be cancelled manually. See CANCEL DECOMMISSION for details
6) Load error hub:
Currently, two types of Hub are supported: Mysql and Broker. You need to specify "type" = "mysql" or "type" = "broker" in PROPERTIES.
If you need to delete the current load error hub, you can set type to null.
1) When using the Mysql type, the error information generated when importing will be inserted into the specified MySQL library table, and then the error information can be viewed directly through the show load warnings statement.

Hub of Mysql type needs to specify the following parameters:
host: mysql host
port: mysql port
user: mysql user
password: mysql password
database mysql database
table: mysql table

2) When the Broker type is used, the error information generated when importing will form a file and be written to the designated remote storage system through the broker. Make sure that the corresponding broker is deployed
Hub of Broker type needs to specify the following parameters:
Broker: Name of broker
Path: Remote Storage Path
Other properties: Other information necessary to access remote storage, such as authentication information.

7) Modify BE node attributes currently supports the following attributes:
1. tag.location：Resource tag
2. disable_query: Query disabled attribute
3. disable_load: Load disabled attribute

8) Remote storage
Currently supports adding object storage (S3, BOS) as remote storage.
`type=s3` needs to be specified in PROPERTIES.
1) When using S3 as remote storage, the following parameters need to be set
s3_endpoint: s3 endpoint
s3_region: s3 region
s3_root_path: s3 root directory
s3_access_key: s3 access key
s3_secret_key: s3 secret key
s3_max_connections: the maximum number of s3 connections, the default is 50
s3_request_timeout_ms: s3 request timeout, in milliseconds, the default is 3000
s3_connection_timeout_ms: s3 connection timeout, in milliseconds, the default is 1000
2) Supports modifying parameter information other than `type`.

## example

1. Add a node
ALTER SYSTEM ADD BACKEND "host:port";

2. Adding an idle node
ALTER SYSTEM ADD FREE BACKEND "host:port";

3. Delete two nodes
ALTER SYSTEM DROP BACKEND "host1:port", "host2:port";

4. offline two nodes
ALTER SYSTEM DECOMMISSION BACKEND "host1:port", "host2:port";

5. Add two Hdfs Broker
ALTER SYSTEM ADD BROKER hdfs "host1:port", "host2:port";

6. Add a load error hub of Mysql type
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "mysql",
"host" = "192.168.1.17"
"port" = "3306",
"User" = "my" name,
"password" = "my_passwd",
"database" = "doris_load",
"table" = "load_errors"
);

7. 添加一个 Broker 类型的 load error hub
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "broker",
"Name" = BOS,
"path" = "bos://backup-cmy/logs",
"bos_endpoint" ="http://gz.bcebos.com",
"bos_accesskey" = "069fc278xxxxxx24ddb522",
"bos_secret_accesskey"="700adb0c6xxxxxx74d59eaa980a"
);

8. Delete the current load error hub
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES
("type"= "null");

9. Modify BE resource tag
ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("tag.location" = "group_a");

10. Modify the query disabled attribute of BE
ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("disable_query" = "true");

11. Modify the load disabled attribute of BE
ALTER SYSTEM MODIFY BACKEND "host1:9050" SET ("disable_load" = "true");

12. Add remote storage
ALTER SYSTEM ADD REMOTE STORAGE remote_s3 PROPERTIES
(
"type" = "s3",
"s3_endpoint" = "bj",
"s3_region" = "bj",
"s3_root_path" = "/path/to/root",
"s3_access_key" = "bbb",
"s3_secret_key" = "aaaa",
"s3_max_connections" = "50",
"s3_request_timeout_ms" = "3000",
"s3_connection_timeout_ms" = "1000"
);

13. Drop remote storage
ALTER SYSTEM DROP REMOTE STORAGE remote_s3;

14. Modify remote storage
ALTER SYSTEM MODIFY REMOTE STORAGE remote_s3 PROPERTIES
(
"s3_access_key" = "bbb",
"s3_secret_key" = "aaaa"
);

## keyword
AGE,SYSTEM,BACKGROUND,BROKER,FREE,REMOTE STORAGE
62 changes: 62 additions & 0 deletions docs/en/sql-reference/sql-statements/Administration/SHOW REMOTE STORAGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
{
"title": "SHOW REMOTE STORAGES",
"language": "en"
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# SHOW REMOTE STORAGES

## Description

This statement is used to view the existing remote storages
Grammar:
SHOW REMOTE STORAGES;

Explain:
1. Name: name of remote storage
2. Type: type of remote storage
3. Properties: properties of reomte storage

## Example

View the remote storage information of the current cluster.

```
mysql> show remote storages;
+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Name | Type | Properties |
+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| remote_s3 | S3 | "s3_secret_key" = "*XXX",
"s3_region" = "bj",
"s3_access_key" = "bbb",
"s3_max_connections" = "50",
"s3_connection_timeout_ms" = "1000",
"s3_root_path" = "/path/to/root",
"s3_endpoint" = "bj",
"s3_request_timeout_ms" = "3000" |
+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```

## keyword

SHOW, REMOTE, REMOTE STORAGES
2 changes: 2 additions & 0 deletions docs/en/sql-reference/sql-statements/Data Definition/ALTER TABLE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@ under the License.
1) The following attributes of the modified partition are currently supported.
- storage_medium
- storage_cooldown_time
- storage_cold_medium
- remote_storage
- replication_num
— in_memory
2) For single-partition tables, partition_name is the same as the table name.
Expand Down
Loading