From f11ceba57e3c2bf956ae2c8800f17cad9df9c19b Mon Sep 17 00:00:00 2001 From: FiatLux Date: Fri, 1 Apr 2022 15:33:50 +0800 Subject: [PATCH 01/36] Update telemetry.md (#8010) --- telemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/telemetry.md b/telemetry.md index 727412ab49a32..ebb09737529d5 100644 --- a/telemetry.md +++ b/telemetry.md @@ -262,4 +262,4 @@ To meet compliance requirements in different countries or regions, the usage inf - For IP addresses from the Chinese mainland, usage information is sent to and stored on cloud servers in the Chinese mainland. - For IP addresses from outside of the Chinese mainland, usage information is sent to and stored on cloud servers in the US. -See [PingCAP Privacy Policy](https://pingcap.com/privacy-policy) for details. +See [PingCAP Privacy Policy](https://en.pingcap.com/privacy-policy/) for details. From 6b92fac8ab5719ac0aa38a8e28a86a7cec499f48 Mon Sep 17 00:00:00 2001 From: xhe Date: Fri, 1 Apr 2022 15:51:26 +0800 Subject: [PATCH 02/36] typo (#8011) --- placement-rules-in-sql.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/placement-rules-in-sql.md b/placement-rules-in-sql.md index c1099b1b1ea92..4fda889f20992 100644 --- a/placement-rules-in-sql.md +++ b/placement-rules-in-sql.md @@ -127,7 +127,7 @@ In addition to the placement options above, you can also use the advance configu | Option Name | Description | | --------------| ------------ | -| `CONSTRAINTS` | A list of constraints that apply to all roles. For example, `CONSTRAINTS="[+disk=ssd]`. | +| `CONSTRAINTS` | A list of constraints that apply to all roles. For example, `CONSTRAINTS="[+disk=ssd]"`. | | `LEADER_CONSTRAINTS` | A list of constraints that only apply to leader. | | `FOLLOWER_CONSTRAINTS` | A list of constraints that only apply to followers. | | `LEARNER_CONSTRAINTS` | A list of constraints that only apply to learners. | @@ -247,7 +247,7 @@ In dictionary format, constraints also indicate a number of instances that apply | Tool Name | Minimum supported version | Description | | --- | --- | --- | -| Backup & Restore (BR) | 6.0 | Supports importing and exporting placement rules. Refer to [BR Compatibility](/br/backup-and-restore-tool.md#compatibility] for details. | +| Backup & Restore (BR) | 6.0 | Supports importing and exporting placement rules. Refer to [BR Compatibility](/br/backup-and-restore-tool.md#compatibility) for details. | | TiDB Lightning | Not compatible yet | | | TiCDC | 6.0 | Ignores placement rules, and does not replicate the rules to the downstream | | TiDB Binlog | 6.0 | Ignores placement rules, and does not replicate the rules to the downstream | From 86911a1300adf409d25d3ef301dbe75b45de9a83 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Fri, 1 Apr 2022 16:30:30 +0800 Subject: [PATCH 03/36] Update the link of privacy policy (#8022) From ed764b3b0cd20c038da6799103819ec3926db875 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 1 Apr 2022 17:50:29 +0800 Subject: [PATCH 04/36] Update docs for base64 encoding of password and kill auto analyze (#7896) --- statistics.md | 28 ++++++++++++++++++- .../tidb-lightning-configuration.md | 3 +- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/statistics.md b/statistics.md index 8438faaca751b..94ef2ba2f0839 100644 --- a/statistics.md +++ b/statistics.md @@ -331,7 +331,33 @@ When the ratio of the number of modified rows to the total number of rows of `tb > > Currently, the automatic update does not record the configuration items input at manual `ANALYZE`. Therefore, when you use the `WITH` syntax to control the collecting behavior of `ANALYZE`, you need to manually set scheduled tasks to collect statistics. -Before v5.0, when you execute a query, TiDB collects feedback with `feedback-probability` and updates the histogram and Count-Min Sketch based on the feedback. **Since v5.0, this feature is disabled by default, and it is not recommended to enable this feature.** +Before TiDB v5.0, when you execute a query, TiDB collects feedback with `feedback-probability` and updates the histogram and Count-Min Sketch based on the feedback. **Since v5.0, this feature is disabled by default, and it is not recommended to enable this feature.** + +Since TiDB v6.0, TiDB supports using the `KILL` statement to terminate an `ANALYZE` task running in the background. If you find that an `ANALYZE` task running in the background consumes a lot of resources and affects your application, you can terminate the `ANALYZE` task by taking the following steps: + +1. Execute the following SQL statement to get the TiDB instance address and the `ID` of the background `ANALYZE` task: + + {{< copyable "sql" >}} + + ```sql + SELECT ci.instance as instance, cp.id as id FROM information_schema.cluster_info ci, information_schema.cluster_processlist cp WHERE ci.status_address = cp.instance and ci.type = 'tidb' and cp.info like 'analyze table %' and cp.user = '' and cp.host = ''; + ``` + + If there is no result, no `ANALYZE` task is running in the background. + +2. Use a client to connect to the TiDB instance where the background `ANALYZE` task is running, and then execute the following `KILL` statement: + + {{< copyable "sql" >}} + + ```sql + KILL TIDB ${id}; + ``` + + `${id}` is the `ID` of the background `ANALYZE` task obtained in the previous step. + + > **Note:** + > + > The above `KILL` statement only works on a TiDB instance that is executing a background `ANALYZE` task. Therefore, you must use a client to connect to that TiDB instance first. If you use a client to connect to another TiDB instance, or if there is a proxy between the client and TiDB, the `KILL` statement cannot terminate the background `ANALYZE` task. For more information, see [`KILL [TIDB]`](/sql-statements/sql-statement-kill.md). ### Control `ANALYZE` concurrency diff --git a/tidb-lightning/tidb-lightning-configuration.md b/tidb-lightning/tidb-lightning-configuration.md index 28a36e469d2e9..38ae9c563793f 100644 --- a/tidb-lightning/tidb-lightning-configuration.md +++ b/tidb-lightning/tidb-lightning-configuration.md @@ -262,6 +262,7 @@ trim-last-separator = false host = "172.16.31.1" port = 4000 user = "root" +# Configure the password to connect to TiDB. The password can either be plaintext or Base64 encoded. password = "" # Table schema information is fetched from TiDB via this status-port. status-port = 10080 @@ -440,7 +441,7 @@ min-available-ratio = 0.05 | --tidb-port *port* | TiDB server port (default = 4000) | `tidb.port` | | --tidb-status *port* | TiDB status port (default = 10080) | `tidb.status-port` | | --tidb-user *user* | User name to connect to TiDB | `tidb.user` | -| --tidb-password *password* | Password to connect to TiDB | `tidb.password` | +| --tidb-password *password* | Password to connect to TiDB. The password can either be plaintext or Base64 encoded. | `tidb.password` | | --no-schema | Ignore schema files, get schema directly from TiDB | `mydumper.no-schema` | | --enable-checkpoint *bool* | Whether to enable checkpoints (default = true) | `checkpoint.enable` | | --analyze *level* | Analyze tables after importing. Available values are "required", "optional" (default value), and "off" | `post-restore.analyze` | From bf23c0a5317fe3150b9c330e8779d8f7d5610e49 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Fri, 1 Apr 2022 19:44:29 +0800 Subject: [PATCH 05/36] update documents related to Plan Cache (#7956) --- sql-prepare-plan-cache.md | 116 ++++++++++++++++++++++++++++++++++++++ system-variables.md | 7 +++ 2 files changed, 123 insertions(+) diff --git a/sql-prepare-plan-cache.md b/sql-prepare-plan-cache.md index 1492325accc5c..ee454bc52abc1 100644 --- a/sql-prepare-plan-cache.md +++ b/sql-prepare-plan-cache.md @@ -119,3 +119,119 @@ MySQL [test]> select @@last_plan_from_cache; +------------------------+ 1 row in set (0.00 sec) ``` + +## Clear execution plan cache + +You can clear execution plan cache by executing the `ADMIN FLUSH [SESSION | INSTANCE] PLAN_CACHE` statement. + +In this statement, `[SESSION | INSTANCE]`specifies whether the plan cache is cleared for the current session or the whole TiDB instance. If the scope is not specified, the statement above applies to the `SESSION` cache by default. + +The following is an example of clearing the `SESSION` execution plan cache: + +{{< copyable "sql" >}} + +```sql +MySQL [test]> create table t (a int); +Query OK, 0 rows affected (0.00 sec) + +MySQL [test]> prepare stmt from 'select * from t'; +Query OK, 0 rows affected (0.00 sec) + +MySQL [test]> execute stmt; +Empty set (0.00 sec) + +MySQL [test]> execute stmt; +Empty set (0.00 sec) + +MySQL [test]> select @@last_plan_from_cache; -- Select the cached plan ++------------------------+ +| @@last_plan_from_cache | ++------------------------+ +| 1 | ++------------------------+ +1 row in set (0.00 sec) + +MySQL [test]> admin flush session plan_cache; -- Clear the cached plan of the current session +Query OK, 0 rows affected (0.00 sec) + +MySQL [test]> execute stmt; +Empty set (0.00 sec) + +MySQL [test]> select @@last_plan_from_cache; -- The cached plan cannot be selected again, because it has been cleared ++------------------------+ +| @@last_plan_from_cache | ++------------------------+ +| 0 | ++------------------------+ +1 row in set (0.00 sec) +``` + +Currently, TiDB does not support clearing `GLOBAL` execution plan cache. That means you cannot clear the cached plan of the whole TiDB cluster. The following error is reported if you try to clear the `GLOBAL` execution plan cache: + +{{< copyable "sql" >}} + +```sql +MySQL [test]> admin flush global plan_cache; +ERROR 1105 (HY000): Do not support the 'admin flush global scope.' +``` + +## Ignore the `COM_STMT_CLOSE` command and the `DEALLOCATE PREPARE` statement + +To reduce the syntax parsing cost of SQL statements, it is recommended that you run `prepare stmt` once, then `execute stmt` multiple times before running `deallocate prepare`: + +{{< copyable "sql" >}} + +```sql +MySQL [test]> prepare stmt from '...'; -- Prepare once +MySQL [test]> execute stmt using ...; -- Execute once +MySQL [test]> ... +MySQL [test]> execute stmt using ...; -- Execute multiple times +MySQL [test]> deallocate prepare stmt; -- Release the prepared statement +``` + +In real practice, you may be used to running `deallocate prepare` each time after running `execute stmt`, as shown below: + +{{< copyable "sql" >}} + +```sql +MySQL [test]> prepare stmt from '...'; -- Prepare once +MySQL [test]> execute stmt using ...; +MySQL [test]> deallocate prepare stmt; -- Release the prepared statement +MySQL [test]> prepare stmt from '...'; -- Prepare twice +MySQL [test]> execute stmt using ...; +MySQL [test]> deallocate prepare stmt; -- Release the prepared statement +``` + +In such practice, the plan obtained by the first executed statement cannot be reused by the second executed statement. + +To address the problem, you can set the system varible [`tidb_ignore_prepared_cache_close_stmt`](/system-variables.md#tidb_ignore_prepared_cache_close_stmt-new-in-v600) to `ON` so TiDB ignores commands to close `prepare stmt`: + +{{< copyable "sql" >}} + +```sql +mysql> set @@tidb_ignore_prepared_cache_close_stmt=1; -- Enable the variable +Query OK, 0 rows affected (0.00 sec) + +mysql> prepare stmt from 'select * from t'; -- Prepare once +Query OK, 0 rows affected (0.00 sec) + +mysql> execute stmt; -- Execute once +Empty set (0.00 sec) + +mysql> deallocate prepare stmt; -- Release after the first execute +Query OK, 0 rows affected (0.00 sec) + +mysql> prepare stmt from 'select * from t'; -- Prepare twice +Query OK, 0 rows affected (0.00 sec) + +mysql> execute stmt; -- Execute twice +Empty set (0.00 sec) + +mysql> select @@last_plan_from_cache; -- Reuse the last plan ++------------------------+ +| @@last_plan_from_cache | ++------------------------+ +| 1 | ++------------------------+ +1 row in set (0.00 sec) +``` \ No newline at end of file diff --git a/system-variables.md b/system-variables.md index bb19b7a353731..0272b17d23595 100644 --- a/system-variables.md +++ b/system-variables.md @@ -1116,6 +1116,13 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - When the parameter of the aggregate function is not distinct, `HashAgg` is run concurrently and respectively in two phases - the `partial` phase and the `final` phase. - A value of `-1` means that the value of `tidb_executor_concurrency` will be used instead. +### tidb_ignore_prepared_cache_close_stmt (New in v6.0.0) + +- Scope: SESSION | GLOBAL +- Default value: `OFF` +- This variable is used to set whether to ignore the commands for closing prepared statement cache. +- When this variable is set to `ON`, the `COM_STMT_CLOSE` command of the Binary protocol and the [`DEALLOCATE PREPARE`](/sql-statements/sql-statement-deallocate.md) statement of the text protocol are ignored. For details, see [Ignore the `COM_STMT_CLOSE` command and the `DEALLOCATE PREPARE` statement](/sql-prepare-plan-cache.md#ignore-the-com_stmt_close-command-and-the-deallocate-prepare-statement). + ### tidb_index_join_batch_size - Scope: SESSION | GLOBAL From 59b3a2bee46e759e6571fd73e104d27af3972d28 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 1 Apr 2022 19:51:49 +0800 Subject: [PATCH 06/36] Add missing example in ALTER PLACEMENT POLICY (#8034) --- sql-statements/sql-statement-alter-placement-policy.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sql-statements/sql-statement-alter-placement-policy.md b/sql-statements/sql-statement-alter-placement-policy.md index 9a13e03798388..f3744ad2364c7 100644 --- a/sql-statements/sql-statement-alter-placement-policy.md +++ b/sql-statements/sql-statement-alter-placement-policy.md @@ -9,6 +9,11 @@ summary: The usage of ALTER PLACEMENT POLICY in TiDB. `ALTER PLACEMENT POLICY` _replaces_ the previous policy with the new definition. It does not _merge_ the old policy with the new one. In the following example, `FOLLOWERS=4` is lost when the `ALTER PLACEMENT POLICY` is executed: +```sql +CREATE PLACEMENT POLICY p1 FOLLOWERS=4; +ALTER PLACEMENT POLICY p1 PRIMARY_REGION="us-east-1" REGIONS="us-east-1,us-west-1"; +``` + ## Synopsis ```ebnf+diagram From 4d59789e71b1d3d4b9ddd945ff76214628f09343 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Fri, 1 Apr 2022 20:16:29 +0800 Subject: [PATCH 07/36] update SPM doc for SQL plan auto capture (#7975) --- sql-plan-management.md | 341 +++++++++++++++++++++++++++++------------ system-variables.md | 9 ++ 2 files changed, 253 insertions(+), 97 deletions(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index ec61c65f47f4c..8f7be10ccd162 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -22,42 +22,46 @@ CREATE [GLOBAL | SESSION] BINDING FOR BindableStmt USING BindableStmt This statement binds SQL execution plans at the GLOBAL or SESSION level. Currently, supported bindable SQL statements (BindableStmt) in TiDB include `SELECT`, `DELETE`, `UPDATE`, and `INSERT` / `REPLACE` with `SELECT` subqueries. +> **Note:** +> +> Bindings have higher priority over manually added hints. Therefore, when you execute a statement containing a hint while a corresponding binding is present, the hint controlling the behavior of the optimizer does not take effect. However, other types of hints are still effective. + Specifically, two types of these statements cannot be bound to execution plans due to syntax conflicts. See the following examples: ```sql --- Type one: Statements that get the Cartesian product by using the `join` keyword and not specifying the associated columns with the `using` keyword. -create global binding for - select * from t t1 join t t2 -using - select * from t t1 join t t2; - --- Type two: `DELETE` statements that contain the `using` keyword. -create global binding for - delete from t1 using t1 join t2 on t1.a = t2.a -using - delete from t1 using t1 join t2 on t1.a = t2.a; +-- Type one: Statements that get the Cartesian product by using the `JOIN` keyword and not specifying the associated columns with the `USING` keyword. +CREATE GLOBAL BINDING for + SELECT * FROM t t1 JOIN t t2 +USING + SELECT * FROM t t1 JOIN t t2; + +-- Type two: `DELETE` statements that contain the `USING` keyword. +CREATE GLOBAL BINDING for + DELETE FROM t1 USING t1 JOIN t2 ON t1.a = t2.a +USING + DELETE FROM t1 USING t1 JOIN t2 ON t1.a = t2.a; ``` You can bypass syntax conflicts by using equivalent statements. For example, you can rewrite the above statements in the following ways: ```sql --- First rewrite of type one statements: Add a `using` clause for the `join` keyword. -create global binding for - select * from t t1 join t t2 using (a) +-- First rewrite of type one statements: Add a `USING` clause for the `JOIN` keyword. +CREATE GLOBAL BINDING for + SELECT * FROM t t1 JOIN t t2 USING (a) +USING + SELECT * FROM t t1 JOIN t t2 USING (a); + +-- Second rewrite of type one statements: Delete the `JOIN` keyword. +CREATE GLOBAL BINDING for + SELECT * FROM t t1, t t2 +USING + SELECT * FROM t t1, t t2; + +-- Rewrite of type two statements: Remove the `USING` keyword from the `delete` statement. +CREATE GLOBAL BINDING for + DELETE t1 FROM t1 JOIN t2 ON t1.a = t2.a using - select * from t t1 join t t2 using (a); - --- Second rewrite of type one statements: Delete the `join` keyword. -create global binding for - select * from t t1, t t2 -using - select * from t t1, t t2; - --- Rewrite of type two statements: Remove the `using` keyword from the `delete` statement. -create global binding for - delete t1 from t1 join t2 on t1.a = t2.a -using - delete t1 from t1 join t2 on t1.a = t2.a; + DELETE t1 FROM t1 JOIN t2 ON t1.a = t2.a; ``` > **Note:** @@ -68,16 +72,16 @@ Here are two examples: ```sql -- The hint takes effect in the following statement. -create global binding for - insert into t1 select * from t2 where a > 1 and b = 1 +CREATE GLOBAL BINDING for + INSERT INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1 using - insert into t1 select /*+ use_index(@sel_1 t2, a) */ * from t2 where a > 1 and b = 1; + INSERT INTO t1 SELECT /*+ use_index(@sel_1 t2, a) */ * FROM t2 WHERE a > 1 AND b = 1; -- The hint cannot take effect in the following statement. -create global binding for - insert into t1 select * from t2 where a > 1 and b = 1 +CREATE GLOBAL BINDING for + INSERT INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1 using - insert /*+ use_index(@sel_1 t2, a) */ into t1 select * from t2 where a > 1 and b = 1; + INSERT /*+ use_index(@sel_1 t2, a) */ INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1; ``` If you do not specify the scope when creating an execution plan binding, the default scope is SESSION. The TiDB optimizer normalizes bound SQL statements and stores them in the system table. When processing SQL queries, if a normalized statement matches one of the bound SQL statements in the system table and the system variable `tidb_use_plan_baselines` is set to `on` (the default value is `on`), TiDB then uses the corresponding optimizer hint for this statement. If there are multiple matchable execution plans, the optimizer chooses the least costly one to bind. @@ -85,9 +89,9 @@ If you do not specify the scope when creating an execution plan binding, the def `Normalization` is a process that converts a constant in an SQL statement to a variable parameter and explicitly specifies the database for tables referenced in the query, with standardized processing on the spaces and line breaks in the SQL statement. See the following example: ```sql -select * from t where a > 1 +SELECT * FROM t WHERE a > 1 -- After normalization, the above statement is as follows: -select * from test . t where a > ? +SELECT * FROM test . t WHERE a > ? ``` > **Note:** @@ -97,15 +101,15 @@ select * from test . t where a > ? > For example: > > ```sql -> select * from t limit 10 -> select * from t limit 10, 20 -> select * from t where a in (1) -> select * from t where a in (1,2,3) +> SELECT * FROM t limit 10 +> SELECT * FROM t limit 10, 20 +> SELECT * FROM t WHERE a IN (1) +> SELECT * FROM t WHERE a IN (1,2,3) > -- After normalization, the above statements are as follows: -> select * from test . t limit ? -> select * from test . t limit ... -> select * from test . t where a in ( ? ) -> select * from test . t where a in ( ... ) +> SELECT * FROM test . t limit ? +> SELECT * FROM test . t limit ... +> SELECT * FROM test . t WHERE a IN ( ? ) +> SELECT * FROM test . t WHERE a IN ( ... ) > ``` > > When bindings are created, TiDB treats SQL statements that contain a single constant and SQL statements that contain multiple constants joined by commas differently. Therefore, you need to create bindings for the two SQL types separately. @@ -116,25 +120,25 @@ For example: ```sql -- Creates a GLOBAL binding and specifies using `sort merge join` in this binding. -create global binding for - select * from t1, t2 where t1.id = t2.id -using - select /*+ merge_join(t1, t2) */ * from t1, t2 where t1.id = t2.id; +CREATE GLOBAL BINDING for + SELECT * FROM t1, t2 WHERE t1.id = t2.id +USING + SELECT /*+ merge_join(t1, t2) */ * FROM t1, t2 WHERE t1.id = t2.id; -- The execution plan of this SQL statement uses the `sort merge join` specified in the GLOBAL binding. -explain select * from t1, t2 where t1.id = t2.id; +explain SELECT * FROM t1, t2 WHERE t1.id = t2.id; -- Creates another SESSION binding and specifies using `hash join` in this binding. -create binding for - select * from t1, t2 where t1.id = t2.id -using - select /*+ hash_join(t1, t2) */ * from t1, t2 where t1.id = t2.id; +CREATE BINDING for + SELECT * FROM t1, t2 WHERE t1.id = t2.id +USING + SELECT /*+ hash_join(t1, t2) */ * FROM t1, t2 WHERE t1.id = t2.id; -- In the execution plan of this statement, `hash join` specified in the SESSION binding is used, instead of `sort merge join` specified in the GLOBAL binding. -explain select * from t1, t2 where t1.id = t2.id; +explain SELECT * FROM t1, t2 WHERE t1.id = t2.id; ``` -When the first `select` statement is being executed, the optimizer adds the `sm_join(t1, t2)` hint to the statement through the binding in the GLOBAL scope. The top node of the execution plan in the `explain` result is MergeJoin. When the second `select` statement is being executed, the optimizer uses the binding in the SESSION scope instead of the binding in the GLOBAL scope and adds the `hash_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is HashJoin. +When the first `SELECT` statement is being executed, the optimizer adds the `sm_join(t1, t2)` hint to the statement through the binding in the GLOBAL scope. The top node of the execution plan in the `explain` result is MergeJoin. When the second `SELECT` statement is being executed, the optimizer uses the binding in the SESSION scope instead of the binding in the GLOBAL scope and adds the `hash_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is HashJoin. Each standardized SQL statement can have only one binding created using `CREATE BINDING` at a time. When multiple bindings are created for the same standardized SQL statement, the last created binding is retained, and all previous bindings (created and evolved) are marked as deleted. But session bindings and global bindings can coexist and are not affected by this logic. @@ -142,13 +146,13 @@ In addition, when you create a binding, TiDB requires that the session is in a d The original SQL statement and the bound statement must have the same text after normalization and hint removal, or the binding will fail. Take the following examples: -- This binding can be created successfully because the texts before and after parameterization and hint removal are the same: `select * from test . t where a > ?` +- This binding can be created successfully because the texts before and after parameterization and hint removal are the same: `SELECT * FROM test . t WHERE a > ?` ```sql CREATE BINDING FOR SELECT * FROM t WHERE a > 1 USING SELECT * FROM t use index (idx) WHERE a > 2 ``` -- This binding will fail because the original SQL statement is processed as `select * from test . t where a > ?`, while the bound SQL statement is processed differently as `select * from test . t where b > ?`. +- This binding will fail because the original SQL statement is processed as `SELECT * FROM test . t WHERE a > ?`, while the bound SQL statement is processed differently as `SELECT * FROM test . t WHERE b > ?`. ```sql CREATE BINDING FOR SELECT * FROM t WHERE a > 1 USING SELECT * FROM t use index(idx) WHERE b > 2 @@ -174,10 +178,10 @@ The following example is based on the example in [create binding](#create-a-bind ```sql -- Drops the binding created in the SESSION scope. -drop session binding for select * from t1, t2 where t1.id = t2.id; +drop session binding for SELECT * FROM t1, t2 WHERE t1.id = t2.id; -- Views the SQL execution plan again. -explain select * from t1,t2 where t1.id = t2.id; +explain SELECT * FROM t1,t2 WHERE t1.id = t2.id; ``` In the example above, the dropped binding in the SESSION scope shields the corresponding binding in the GLOBAL scope. The optimizer does not add the `sm_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is not fixed to MergeJoin by this hint. Instead, the top node is independently selected by the optimizer according to the cost estimation. @@ -186,7 +190,19 @@ In the example above, the dropped binding in the SESSION scope shields the corre > > Executing `DROP GLOBAL BINDING` drops the binding in the current tidb-server instance cache and changes the status of the corresponding row in the system table to 'deleted'. This statement does not directly delete the records in the system table, because other tidb-server instances need to read the 'deleted' status to drop the corresponding binding in their cache. For the records in these system tables with the status of 'deleted', at every 100 `bind-info-lease` (the default value is `3s`, and `300s` in total) interval, the background thread triggers an operation of reclaiming and clearing on the bindings of `update_time` before 10 `bind-info-lease` (to ensure that all tidb-server instances have read the 'deleted' status and updated the cache). -### View binding +## Change binding status + +{{< copyable "sql" >}} + +```sql +SET BINDING [ENABLED | DISABLED] FOR BindableStmt; +``` + +You can execute this statement to change the status of a binding. The default status is ENABLED. The effective scope is GLOBAL by default and cannot be modified. + +When executing this statement, you can only change the status of a binding from `Disabled` to `Enabled` or from `Enabled` to `Disabled`. If no binding is available for status changes, a warning message is returned, saying `There are no bindings can be set the status. Please check the SQL text`. Note that a binding in `Disabled` status is not used by any query. + +### View bindings {{< copyable "sql" >}} @@ -201,41 +217,98 @@ This statement outputs the execution plan bindings at the GLOBAL or SESSION leve | original_sql | Original SQL statement after parameterization | | bind_sql | Bound SQL statement with hints | | default_db | Default database | -| status | Status including Using, Deleted, Invalid, Rejected, and Pending verification| +| status | Status including Enabled (replacing the Using status from v6.0), Disabled, Deleted, Invalid, Rejected, and Pending verification| | create_time | Creating time | | update_time | Updating time | | charset | Character set | | collation | Ordering rule | | source | The way in which a binding is created, including `manual` (created by the `create [global] binding` SQL statement), `capture` (captured automatically by TiDB), and `evolve` (evolved automatically by TiDB) | -### Troubleshoot binding +### Troubleshoot a binding -{{< copyable "sql" >}} +You can use either of the following methods to troubleshoot a binding: -```sql -SELECT @@[SESSION.]last_plan_from_binding; -``` +- Use the system variable [`last_plan_from_binding`](/system-variables.md#last_plan_from_binding-new-in-v40) to show whether the execution plan used by the last executed statement is from the binding. -This statement uses the system variable [`last_plan_from_binding`](/system-variables.md#last_plan_from_binding-new-in-v40) to show whether the execution plan used by the last executed statement is from the binding. + {{< copyable "sql" >}} -In addition, when you use the `explain format = 'verbose'` statement to view the query plan of a SQL statement, if the SQL statement uses binding, the `explain` statement will return a warning. In this situation, you can check the warning message to learn which binding is used in the SQL statement. + ```sql + -- Create a global binding + CREATE GLOBAL BINDING for + SELECT * FROM t + USING + SELECT /*+ USE_INDEX(t, idx_a) */ * FROM t; + + SELECT * FROM t; + SELECT @@[SESSION.]last_plan_from_binding; + ``` -```sql --- Create a global binding. + ```sql + +--------------------------+ + | @@last_plan_from_binding | + +--------------------------+ + | 1 | + +--------------------------+ + 1 row in set (0.00 sec) + ``` -create global binding for - select * from t -using - select * from t; +- Use the `explain format = 'verbose'` statement to view the query plan of a SQL statement. If the SQL statement uses a binding, you can run `show warnings` to check which binding is used in the SQL statement. + + ```sql + -- Create a global binding + + CREATE GLOBAL BINDING for + SELECT * FROM t + USING + SELECT /*+ USE_INDEX(t, idx_a) */ * FROM t; + + -- Use explain format = 'verbose' to view the execution plan of a SQL statement + + explain format = 'verbose' SELECT * FROM t; + + -- Run `show warnings` to view the binding used in the query. + + show warnings; + ``` + + ```sql + +-------+------+--------------------------------------------------------------------------+ + | Level | Code | Message | + +-------+------+--------------------------------------------------------------------------+ + | Note | 1105 | Using the bindSQL: SELECT /*+ USE_INDEX(`t` `idx_a`)*/ * FROM `test`.`t` | + +-------+------+--------------------------------------------------------------------------+ + 1 row in set (0.01 sec) + + ``` + +### Cache bindings + +Each TiDB instance has a least recently used (LRU) cache for bindings. The cache capacity is controlled by the system variable [`tidb_mem_quota_binding_cache`](/system-variables.md#tidb_mem_quota_binding_cache-new-in-v600). You can view bindings that are cached in the TiDB instance. + +To view the cache status of bindings, run the `SHOW binding_cache status` statement. In this statement, the effective scope is GLOBAL by default and cannot be modified. This statement returns the number of available bindings in the cache, the total number of available bindings in the system, memory usage of all cached bindings, and the total memory for the cache. + +{{< copyable "sql" >}} + +```sql --- Use the `explain format = 'verbose'` statement to check the SQL execution plan. Check the warning message to view the binding used in the query. +SHOW binding_cache status; +``` -explain format = 'verbose' select * from t; -show warnings; +```sql ++-------------------+-------------------+--------------+--------------+ +| bindings_in_cache | bindings_in_table | memory_usage | memory_quota | ++-------------------+-------------------+--------------+--------------+ +| 1 | 1 | 159 Bytes | 64 MB | ++-------------------+-------------------+--------------+--------------+ +1 row in set (0.00 sec) ``` ## Baseline capturing +Used for [preventing regression of execution plans during an upgrade](#prevent-regression-of-execution-plans-during-an-upgrade), this feature captures queries that meet capturing conditions and creates bindings for these queries. + +### Enable capturing + To enable baseline capturing, set `tidb_capture_plan_baselines` to `on`. The default value is `off`. > **Note:** @@ -248,7 +321,12 @@ However, TiDB does not automatically capture bindings for the following types of - `EXPLAIN` and `EXPLAIN ANALYZE` statements. - SQL statements executed internally in TiDB, such as `SELECT` queries used for automatically loading statistical information. -- SQL statements that are bound to a manually created execution plan. +- Statements that contain `Enabled` or `Disabled` bindings. +- Statements that are filtered out by capturing conditions. + +> **Note:** +> +> Currently, a binding generates a group of hints to fix an execution plan generated by a query statement. In this way, for the same query, the execution plan does not change. For most OLTP queries, including queries using the same index or Join algorithm (such as HashJoin and IndexJoin), TiDB guarantees plan consistency before and after the binding. However, due to the limitations of hints, TiDB cannot guarantee plan consistency for some complex queries, such as Join of more than two tables, MPP queries, and complex OLAP queries. For `PREPARE` / `EXECUTE` statements and for queries executed with binary protocols, TiDB automatically captures bindings for the real query statements, not for the `PREPARE` / `EXECUTE` statements. @@ -256,6 +334,75 @@ For `PREPARE` / `EXECUTE` statements and for queries executed with binary protoc > > Because TiDB has some embedded SQL statements to ensure the correctness of some features, baseline capturing by default automatically shields these SQL statements. +### Filter out bindings + +This feature allows you to configure a blocklist to filter out queries whose bindings you do not want to capture. A blocklist has three dimensions, table name, frequency, and user name. + +#### Usage + +Insert filtering conditions into the system table `mysql.capture_plan_baselines_blacklist`. Then the filtering conditions take effect in the entire cluster immediately. + +```sql +-- Filter by table name + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'test.t'); + +-- Filter by database name and table name through wildcards + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'test.table_*'); + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'db_*.table_*'); + +-- Filter by frequency + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('frequency', '2'); + +-- Filter by user name + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('user', 'user1'); +``` + +| **Dimension name** | **Description** | Remarks | +| :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | +| table | Filter by table name. Each filtering rule is in the `db.table` format. The supported filtering syntax includes [Plain table names](/table-filter.md#plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If a table name contains illegal characters, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | +| frequency | Filter by frequency. SQL statements executed more than once are captured by default. You can set a high frequency to capture statements that are frequently executed. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | +| user | Filter by user name. Statements executed by blocklisted users are not captured. | If multiple users execute the same statement and their user names are all in the blocklist, this statement is not captured. | + +> **Note:** +> +> - Modifying a blocklist requires the super privilege. +> +> - If a blocklist contains illegal filters, TiDB returns the warning message `[sql-bind] unknown capture filter type, ignore it` in the log. + +### Prevent regression of execution plans during an upgrade + + Before upgrading a TiDB cluster, you can use baseline capturing to prevent regression of execution plans by performing the following steps: + +1. Enable baseline capturing and keep it working for a period of time. + + > **Note:** + > + > Test data shows that long-term working of baseline capturing has a slight impact on the performance of the cluster load. Therefore, it is recommended to enable baseline capturing as long as possible so that important plans (appear twice or above) are captured. + +2. Upgrade the TiDB cluster. After the upgrade, TiDB uses those captured bindings to ensure execution plan consistency. + +3. After the upgrade, delete bindings as required. + + - Check the binding source by running the [`SHOW GLOBAL BINDINGS`](#view-bindings) statement. + + In the output, check the `Source` field to see whether a binding is captured (`capture`) or manually created (`manual`). + + - Determine whether to retain the captured bindings: + + ``` + -- View the plan with the binding enabled + SET @@SESSION.TIDB_USE_PLAN_BASELINES = true; + EXPLAIN FORMAT='VERBOSE' SELECT * FROM t1 WHERE ...; + + -- View the plan with the binding disabled + SET @@SESSION.TIDB_USE_PLAN_BASELINES = false; + EXPLAIN FORMAT='VERBOSE' SELECT * FROM t1 WHERE ...; + ``` + + - If the execution plan is consistent, you can delete the binding safely. + + - If the execution plan is inconsistent, you need to identify the cause, for example, by checking statistics. In this case, you need to retain the binding to ensure plan consistency. + ## Baseline evolution Baseline evolution is an important feature of SPM introduced in TiDB v4.0. @@ -271,7 +418,7 @@ Use the following statement to enable automatic binding evolution: {{< copyable "sql" >}} ```sql -set global tidb_evolve_plan_baselines = on; +SET GLOBAL tidb_evolve_plan_baselines = ON; ``` The default value of `tidb_evolve_plan_baselines` is `off`. @@ -288,7 +435,7 @@ Assume that table `t` is defined as follows: {{< copyable "sql" >}} ```sql -create table t(a int, b int, key(a), key(b)); +CREATE TABLE t(a INT, b INT, KEY(a), KEY(b)); ``` Perform the following query on table `t`: @@ -296,7 +443,7 @@ Perform the following query on table `t`: {{< copyable "sql" >}} ```sql -select * from t where a < 100 and b < 100; +SELECT * FROM t WHERE a < 100 AND b < 100; ``` In the table defined above, few rows meet the `a < 100` condition. But for some reason, the optimizer mistakenly selects the full table scan instead of the optimal execution plan that uses index `a`. You can first use the following statement to create a binding: @@ -304,7 +451,7 @@ In the table defined above, few rows meet the `a < 100` condition. But for some {{< copyable "sql" >}} ```sql -create global binding for select * from t where a < 100 and b < 100 using select * from t use index(a) where a < 100 and b < 100; +CREATE GLOBAL BINDING for SELECT * FROM t WHERE a < 100 AND b < 100 using SELECT * FROM t use index(a) WHERE a < 100 AND b < 100; ``` When the query above is executed again, the optimizer selects index `a` (influenced by the binding created above) to reduce the query time. @@ -343,18 +490,18 @@ Because the baseline evolution automatically creates a new binding, when the que During cluster upgrade, SQL Plan Management (SPM) might cause compatibility issues and make the upgrade fail. To ensure a successful upgrade, you need to include the following list for upgrade precheck: -* When you upgrade from a version earlier than v5.2.0 (that is, v4.0, v5.0, and v5.1) to the current version, make sure that `tidb_evolve_plan_baselines` is disabled before the upgrade. To disable this variable, perform the following steps. +* When you upgrade from a version earlier than v5.2.0 (that is, v4.0, v5.0, and v5.1) to the current version, make sure that `tidb_evolve_plan_baselines` is disabled before the upgrade. To disable this variable, perform the following steps. {{< copyable "sql" >}} ```sql - -- Check whether `tidb_evolve_plan_baselines` is disabled in the earlier version. - - select @@global.tidb_evolve_plan_baselines; - - -- If `tidb_evolve_plan_baselines` is still enabled, disable it. - - set global tidb_evolve_plan_baselines = off; + -- Check whether `tidb_evolve_plan_baselines` is disabled in the earlier version. + + SELECT @@global.tidb_evolve_plan_baselines; + + -- If `tidb_evolve_plan_baselines` is still enabled, disable it. + + SET GLOBAL tidb_evolve_plan_baselines = OFF; ``` * Before you upgrade from v4.0 to the current version, you need to check whether the syntax of all queries corresponding to the available SQL bindings is correct in the new version. If any syntax errors exist, delete the corresponding SQL binding. To do that, perform the following steps. @@ -363,15 +510,15 @@ During cluster upgrade, SQL Plan Management (SPM) might cause compatibility issu ```sql -- Check the query corresponding to the available SQL binding in the version to be upgraded. - - select bind_sql from mysql.bind_info where status = 'using'; - - -- Verify the result from the above SQL query in the test environment of the new version. - + + SELECT bind_sql FROM mysql.bind_info WHERE status = 'using'; + + -- Verify the result from the above SQL query in the test environment of the new version. + bind_sql_0; bind_sql_1; ... - - -- In the case of a syntax error (ERROR 1064 (42000): You have an error in your SQL syntax), delete the corresponding binding. - -- For any other errors (for example, tables are not found), it means that the syntax is compatible. No other operation is needed. + + -- In the case of a syntax error (ERROR 1064 (42000): You have an error in your SQL syntax), delete the corresponding binding. + -- For any other errors (for example, tables are not found), it means that the syntax is compatible. No other operation is needed. ``` diff --git a/system-variables.md b/system-variables.md index 0272b17d23595..dea308d39c802 100644 --- a/system-variables.md +++ b/system-variables.md @@ -1229,6 +1229,15 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - This variable is used to set the memory usage threshold of the local cache in the `Apply` operator. - The local cache in the `Apply` operator is used to speed up the computation of the `Apply` operator. You can set the variable to `0` to disable the `Apply` cache feature. +### tidb_mem_quota_binding_cache (New in v6.0.0) + +- Scope: GLOBAL +- Default value: `67108864` (64 MiB) +- Range: `[0, 2147483647]` +- Unit: Bytes +- This variable is used to set the threshold of the memory used for caching bindings. +- If a system creates or captures excessive bindings, resulting in overuse of memory space, TiDB returns a warning in the log. In this case, the cache cannot hold all available bindings or determine which bindings to store. For this reason, some queries might miss their bindings. To address this problem, you can increase the value of this variable, which increases the memory used for caching bindings. After modifying this parameter, you need to run `admin reload bindings` to reload bindings and validate the modification. + ### tidb_mem_quota_query - Scope: SESSION From 185cdba3dea0f73e86f57cb87c0962e38f4a151a Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Mon, 28 Mar 2022 19:48:59 +0800 Subject: [PATCH 08/36] update SPM doc for SQL plan auto capture --- sql-plan-management.md | 339 +++++++++++++++++++++++++++++------------ system-variables.md | 9 ++ 2 files changed, 252 insertions(+), 96 deletions(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index ec61c65f47f4c..0956696c6b371 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -22,42 +22,46 @@ CREATE [GLOBAL | SESSION] BINDING FOR BindableStmt USING BindableStmt This statement binds SQL execution plans at the GLOBAL or SESSION level. Currently, supported bindable SQL statements (BindableStmt) in TiDB include `SELECT`, `DELETE`, `UPDATE`, and `INSERT` / `REPLACE` with `SELECT` subqueries. +> **Note:** +> +> Bindings have higher priority over hints added manually. Therefore, when you execute a statement containing a hint while a binding is present, the hint controlling the behavior of the optimizer does not take effect. However, other types of hints are still effective. + Specifically, two types of these statements cannot be bound to execution plans due to syntax conflicts. See the following examples: ```sql --- Type one: Statements that get the Cartesian product by using the `join` keyword and not specifying the associated columns with the `using` keyword. -create global binding for - select * from t t1 join t t2 -using - select * from t t1 join t t2; - --- Type two: `DELETE` statements that contain the `using` keyword. -create global binding for - delete from t1 using t1 join t2 on t1.a = t2.a -using - delete from t1 using t1 join t2 on t1.a = t2.a; +-- Type one: Statements that get the Cartesian product by using the `JOIN` keyword and not specifying the associated columns with the `USING` keyword. +CREATE GLOBAL BINDING for + SELECT * FROM t t1 JOIN t t2 +USING + SELECT * FROM t t1 JOIN t t2; + +-- Type two: `DELETE` statements that contain the `USING` keyword. +CREATE GLOBAL BINDING for + DELETE FROM t1 USING t1 JOIN t2 ON t1.a = t2.a +USING + DELETE FROM t1 USING t1 JOIN t2 ON t1.a = t2.a; ``` You can bypass syntax conflicts by using equivalent statements. For example, you can rewrite the above statements in the following ways: ```sql --- First rewrite of type one statements: Add a `using` clause for the `join` keyword. -create global binding for - select * from t t1 join t t2 using (a) +-- First rewrite of type one statements: Add a `USING` clause for the `JOIN` keyword. +CREATE GLOBAL BINDING for + SELECT * FROM t t1 JOIN t t2 USING (a) +USING + SELECT * FROM t t1 JOIN t t2 USING (a); + +-- Second rewrite of type one statements: Delete the `JOIN` keyword. +CREATE GLOBAL BINDING for + SELECT * FROM t t1, t t2 +USING + SELECT * FROM t t1, t t2; + +-- Rewrite of type two statements: Remove the `USING` keyword from the `delete` statement. +CREATE GLOBAL BINDING for + DELETE t1 FROM t1 JOIN t2 ON t1.a = t2.a using - select * from t t1 join t t2 using (a); - --- Second rewrite of type one statements: Delete the `join` keyword. -create global binding for - select * from t t1, t t2 -using - select * from t t1, t t2; - --- Rewrite of type two statements: Remove the `using` keyword from the `delete` statement. -create global binding for - delete t1 from t1 join t2 on t1.a = t2.a -using - delete t1 from t1 join t2 on t1.a = t2.a; + DELETE t1 FROM t1 JOIN t2 ON t1.a = t2.a; ``` > **Note:** @@ -68,16 +72,16 @@ Here are two examples: ```sql -- The hint takes effect in the following statement. -create global binding for - insert into t1 select * from t2 where a > 1 and b = 1 +CREATE GLOBAL BINDING for + INSERT INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1 using - insert into t1 select /*+ use_index(@sel_1 t2, a) */ * from t2 where a > 1 and b = 1; + INSERT INTO t1 SELECT /*+ use_index(@sel_1 t2, a) */ * FROM t2 WHERE a > 1 AND b = 1; -- The hint cannot take effect in the following statement. -create global binding for - insert into t1 select * from t2 where a > 1 and b = 1 +CREATE GLOBAL BINDING for + INSERT INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1 using - insert /*+ use_index(@sel_1 t2, a) */ into t1 select * from t2 where a > 1 and b = 1; + INSERT /*+ use_index(@sel_1 t2, a) */ INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1; ``` If you do not specify the scope when creating an execution plan binding, the default scope is SESSION. The TiDB optimizer normalizes bound SQL statements and stores them in the system table. When processing SQL queries, if a normalized statement matches one of the bound SQL statements in the system table and the system variable `tidb_use_plan_baselines` is set to `on` (the default value is `on`), TiDB then uses the corresponding optimizer hint for this statement. If there are multiple matchable execution plans, the optimizer chooses the least costly one to bind. @@ -85,9 +89,9 @@ If you do not specify the scope when creating an execution plan binding, the def `Normalization` is a process that converts a constant in an SQL statement to a variable parameter and explicitly specifies the database for tables referenced in the query, with standardized processing on the spaces and line breaks in the SQL statement. See the following example: ```sql -select * from t where a > 1 +SELECT * FROM t WHERE a > 1 -- After normalization, the above statement is as follows: -select * from test . t where a > ? +SELECT * FROM test . t WHERE a > ? ``` > **Note:** @@ -97,15 +101,15 @@ select * from test . t where a > ? > For example: > > ```sql -> select * from t limit 10 -> select * from t limit 10, 20 -> select * from t where a in (1) -> select * from t where a in (1,2,3) +> SELECT * FROM t limit 10 +> SELECT * FROM t limit 10, 20 +> SELECT * FROM t WHERE a IN (1) +> SELECT * FROM t WHERE a IN (1,2,3) > -- After normalization, the above statements are as follows: -> select * from test . t limit ? -> select * from test . t limit ... -> select * from test . t where a in ( ? ) -> select * from test . t where a in ( ... ) +> SELECT * FROM test . t limit ? +> SELECT * FROM test . t limit ... +> SELECT * FROM test . t WHERE a IN ( ? ) +> SELECT * FROM test . t WHERE a IN ( ... ) > ``` > > When bindings are created, TiDB treats SQL statements that contain a single constant and SQL statements that contain multiple constants joined by commas differently. Therefore, you need to create bindings for the two SQL types separately. @@ -116,25 +120,25 @@ For example: ```sql -- Creates a GLOBAL binding and specifies using `sort merge join` in this binding. -create global binding for - select * from t1, t2 where t1.id = t2.id -using - select /*+ merge_join(t1, t2) */ * from t1, t2 where t1.id = t2.id; +CREATE GLOBAL BINDING for + SELECT * FROM t1, t2 WHERE t1.id = t2.id +USING + SELECT /*+ merge_join(t1, t2) */ * FROM t1, t2 WHERE t1.id = t2.id; -- The execution plan of this SQL statement uses the `sort merge join` specified in the GLOBAL binding. -explain select * from t1, t2 where t1.id = t2.id; +explain SELECT * FROM t1, t2 WHERE t1.id = t2.id; -- Creates another SESSION binding and specifies using `hash join` in this binding. -create binding for - select * from t1, t2 where t1.id = t2.id -using - select /*+ hash_join(t1, t2) */ * from t1, t2 where t1.id = t2.id; +CREATE BINDING for + SELECT * FROM t1, t2 WHERE t1.id = t2.id +USING + SELECT /*+ hash_join(t1, t2) */ * FROM t1, t2 WHERE t1.id = t2.id; -- In the execution plan of this statement, `hash join` specified in the SESSION binding is used, instead of `sort merge join` specified in the GLOBAL binding. -explain select * from t1, t2 where t1.id = t2.id; +explain SELECT * FROM t1, t2 WHERE t1.id = t2.id; ``` -When the first `select` statement is being executed, the optimizer adds the `sm_join(t1, t2)` hint to the statement through the binding in the GLOBAL scope. The top node of the execution plan in the `explain` result is MergeJoin. When the second `select` statement is being executed, the optimizer uses the binding in the SESSION scope instead of the binding in the GLOBAL scope and adds the `hash_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is HashJoin. +When the first `SELECT` statement is being executed, the optimizer adds the `sm_join(t1, t2)` hint to the statement through the binding in the GLOBAL scope. The top node of the execution plan in the `explain` result is MergeJoin. When the second `SELECT` statement is being executed, the optimizer uses the binding in the SESSION scope instead of the binding in the GLOBAL scope and adds the `hash_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is HashJoin. Each standardized SQL statement can have only one binding created using `CREATE BINDING` at a time. When multiple bindings are created for the same standardized SQL statement, the last created binding is retained, and all previous bindings (created and evolved) are marked as deleted. But session bindings and global bindings can coexist and are not affected by this logic. @@ -142,13 +146,13 @@ In addition, when you create a binding, TiDB requires that the session is in a d The original SQL statement and the bound statement must have the same text after normalization and hint removal, or the binding will fail. Take the following examples: -- This binding can be created successfully because the texts before and after parameterization and hint removal are the same: `select * from test . t where a > ?` +- This binding can be created successfully because the texts before and after parameterization and hint removal are the same: `SELECT * FROM test . t WHERE a > ?` ```sql CREATE BINDING FOR SELECT * FROM t WHERE a > 1 USING SELECT * FROM t use index (idx) WHERE a > 2 ``` -- This binding will fail because the original SQL statement is processed as `select * from test . t where a > ?`, while the bound SQL statement is processed differently as `select * from test . t where b > ?`. +- This binding will fail because the original SQL statement is processed as `SELECT * FROM test . t WHERE a > ?`, while the bound SQL statement is processed differently as `SELECT * FROM test . t WHERE b > ?`. ```sql CREATE BINDING FOR SELECT * FROM t WHERE a > 1 USING SELECT * FROM t use index(idx) WHERE b > 2 @@ -174,10 +178,10 @@ The following example is based on the example in [create binding](#create-a-bind ```sql -- Drops the binding created in the SESSION scope. -drop session binding for select * from t1, t2 where t1.id = t2.id; +drop session binding for SELECT * FROM t1, t2 WHERE t1.id = t2.id; -- Views the SQL execution plan again. -explain select * from t1,t2 where t1.id = t2.id; +explain SELECT * FROM t1,t2 WHERE t1.id = t2.id; ``` In the example above, the dropped binding in the SESSION scope shields the corresponding binding in the GLOBAL scope. The optimizer does not add the `sm_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is not fixed to MergeJoin by this hint. Instead, the top node is independently selected by the optimizer according to the cost estimation. @@ -186,6 +190,18 @@ In the example above, the dropped binding in the SESSION scope shields the corre > > Executing `DROP GLOBAL BINDING` drops the binding in the current tidb-server instance cache and changes the status of the corresponding row in the system table to 'deleted'. This statement does not directly delete the records in the system table, because other tidb-server instances need to read the 'deleted' status to drop the corresponding binding in their cache. For the records in these system tables with the status of 'deleted', at every 100 `bind-info-lease` (the default value is `3s`, and `300s` in total) interval, the background thread triggers an operation of reclaiming and clearing on the bindings of `update_time` before 10 `bind-info-lease` (to ensure that all tidb-server instances have read the 'deleted' status and updated the cache). +## Change binding status + +{{< copyable "sql" >}} + +```sql +SET BINDING [ENABLED | DISABLED] FOR BindableStmt; +``` + +You can execute this statement to change the status of a binding. The default status is ENABLED. The effective scope is GLOBAL and cannot be modified. + +When executing this statement, you can only change the status of a binding from `Disabled` to `Enabled` or from `Enabled` to `Disabled`. If there is no binding for you to change the status, a warning message is returned, saying `There are no bindings can be set the status. Please check the SQL text`. Note that a binding in `Disabled` status is not used by any query. + ### View binding {{< copyable "sql" >}} @@ -201,41 +217,98 @@ This statement outputs the execution plan bindings at the GLOBAL or SESSION leve | original_sql | Original SQL statement after parameterization | | bind_sql | Bound SQL statement with hints | | default_db | Default database | -| status | Status including Using, Deleted, Invalid, Rejected, and Pending verification| +| status | Status including Enabled (available, replacing Using from v6.0), Disabled (unavailable), Deleted, Invalid, Rejected, and Pending verification| | create_time | Creating time | | update_time | Updating time | | charset | Character set | | collation | Ordering rule | | source | The way in which a binding is created, including `manual` (created by the `create [global] binding` SQL statement), `capture` (captured automatically by TiDB), and `evolve` (evolved automatically by TiDB) | -### Troubleshoot binding +### Troubleshoot bindings -{{< copyable "sql" >}} +You can use either of the following methods to troubleshoot bindings: -```sql -SELECT @@[SESSION.]last_plan_from_binding; -``` +- Use the system variable [`last_plan_from_binding`](/system-variables.md#last_plan_from_binding-new-in-v40) to show whether the execution plan used by the last executed statement is from the binding. -This statement uses the system variable [`last_plan_from_binding`](/system-variables.md#last_plan_from_binding-new-in-v40) to show whether the execution plan used by the last executed statement is from the binding. + {{< copyable "sql" >}} -In addition, when you use the `explain format = 'verbose'` statement to view the query plan of a SQL statement, if the SQL statement uses binding, the `explain` statement will return a warning. In this situation, you can check the warning message to learn which binding is used in the SQL statement. + ```sql + -- Create a global binding + CREATE GLOBAL BINDING for + SELECT * FROM t + USING + SELECT /*+ USE_INDEX(t, idx_a) */ * FROM t; + + SELECT * FROM t; + SELECT @@[SESSION.]last_plan_from_binding; + ``` -```sql --- Create a global binding. + ```sql + +--------------------------+ + | @@last_plan_from_binding | + +--------------------------+ + | 1 | + +--------------------------+ + 1 row in set (0.00 sec) + ``` -create global binding for - select * from t -using - select * from t; +- Use the `explain format = 'verbose'` statement to view the query plan of a SQL statement. If the SQL statement uses a binding, you can run `show warnings` to check which binding is used in the SQL statement. + + ```sql + -- Create a global binding + + CREATE GLOBAL BINDING for + SELECT * FROM t + USING + SELECT /*+ USE_INDEX(t, idx_a) */ * FROM t; + + -- Use explain format = 'verbose' to view the execution plan of a SQL statement + + explain format = 'verbose' SELECT * FROM t; + + -- Run `show warnings` to view the binding used in the query. + + show warnings; + ``` + + ```sql + +-------+------+--------------------------------------------------------------------------+ + | Level | Code | Message | + +-------+------+--------------------------------------------------------------------------+ + | Note | 1105 | Using the bindSQL: SELECT /*+ USE_INDEX(`t` `idx_a`)*/ * FROM `test`.`t` | + +-------+------+--------------------------------------------------------------------------+ + 1 row in set (0.01 sec) + + ``` + +### Cache bindings + +Each TiDB instace has a least recently used (LRU) cache for bindings. The cache capacity is controlled by the system variable [`tidb_mem_quota_binding_cache`](/system-variables.md#tidb_mem_quota_binding_cache-new-in-v60). You can view bindings that are cached in the TiDB instance. + +To view the cache status of bindings, run the `SHOW binding_cache status` statement. In this statement, the effective scope is GLOBAL by default and cannot be modified. This statement returns the number of available bindings in the cache, the total number of available bindings in the system, memory usage of all cached bindings, and the memory quota. + +{{< copyable "sql" >}} + +```sql --- Use the `explain format = 'verbose'` statement to check the SQL execution plan. Check the warning message to view the binding used in the query. +SHOW binding_cache status; +``` -explain format = 'verbose' select * from t; -show warnings; +```sql ++-------------------+-------------------+--------------+--------------+ +| bindings_in_cache | bindings_in_table | memory_usage | memory_quota | ++-------------------+-------------------+--------------+--------------+ +| 1 | 1 | 159 Bytes | 64 MB | ++-------------------+-------------------+--------------+--------------+ +1 row in set (0.00 sec) ``` ## Baseline capturing +Used for [preventing rollback of execution plans during an upgrade](#prevent-rollback-of-execution-plans-during-an-upgrade), this function captures queries that meet capturing conditions and create bindings for them. + +### Enable capturing + To enable baseline capturing, set `tidb_capture_plan_baselines` to `on`. The default value is `off`. > **Note:** @@ -248,7 +321,12 @@ However, TiDB does not automatically capture bindings for the following types of - `EXPLAIN` and `EXPLAIN ANALYZE` statements. - SQL statements executed internally in TiDB, such as `SELECT` queries used for automatically loading statistical information. -- SQL statements that are bound to a manually created execution plan. +- Statements that contain `Enabled` or `Disabled` bindings. +- Statements that meet capturing conditions. + +> **Note:** +> +> Existing bindings are fixed by generating a group of hints. In this way, execution plans generated by query statements do not change. For most OLTP queries, including queries using the same index or Join mode (such as HashJoin and IndexJoin), TiDB guarantees plan consistency before and after binding. However, due to limitations of hints, TiDB cannot guarantee plan consistency for some complex queries, such as Join of more than two tables, MPP queries, and complex OLAP queries. For `PREPARE` / `EXECUTE` statements and for queries executed with binary protocols, TiDB automatically captures bindings for the real query statements, not for the `PREPARE` / `EXECUTE` statements. @@ -256,6 +334,75 @@ For `PREPARE` / `EXECUTE` statements and for queries executed with binary protoc > > Because TiDB has some embedded SQL statements to ensure the correctness of some features, baseline capturing by default automatically shields these SQL statements. +### Filter out bindings + +This function allows you to configure a blocklist to filter out queries that meet the requirements. A blocklist has three dimensions, table name, frequency, and user name. + +#### Usage + +Insert filtering conditions into the system table `mysql.capture_plan_baselines_blacklist`. Then the filtering conditions take effect in the entire cluster immediately. + +```sql +-- Filter by table name + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'test.t'); + +-- Filter by database name and table name through wildcards + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'test.table_*'); + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'db_*.table_*'); + +-- Filter by frequency + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('frequency', '2'); + +-- Filter by user name + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('user', 'user1'); +``` + +| ** Dimension ** | ** Dimension name ** | ** Description ** | Remarks | +| :----------- | :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | +| Table name | table | Filter by table name. Each filtering rule is in the form of `db.table` and supports wildcards. For details about the rules, see [Plain table names](/table-filter.md# plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If the table name contains illegal content, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | +| Frequency | frequency | SQL statements executed more than once are captured by default. You can set a high frequency to capture statements more frequently. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | +| User name | user | Statements executed by blocklisted users are not captured. | If multiple users execute the same statement, this statement is not captured only if the user names are all in the blocklist. | + +> **Note:** +> +> - Modifying a blocklist requires the super privilege. +> +> - If the blocklist contains illegal filter content, TiDB returns warning message `[sql-bind] unknown capture filter type, ignore it` in the log. + +### Prevent rollback of execution plans during an upgrade + +When upgrading a TiDB cluster, you can use baseline capturing to prevent rollback of execution plans by performing the following steps: + +1. Enable baseline capturing and keep it working for a period of time. This ensures that most important plans are captured. + + > **Note:** + > + > Test data shows that long-term working of baseline capturing does not affect performance of the cluster load. Therefore, it is recommended to enable baseline capturing as long as possible. + +2. Upgrade the TiDB cluster. After the upgrade, captured bindings work to ensure plan consistency. + +3. After the upgrade, delete bindings as required. + + - Check the binding source by running the [`SHOW BINDINGS`](#view-binding) statement. + + Based on the `Source` field in the output, decide whether the bindings are captured ( `capture`) or manually created (`manual`). + + - Determine whether to retain the captured bindings: + + ``` + -- View the plan with the binding enabled + SET @@SESSION.TIDB_USE_PLAN_BASELINES = true; + EXPLAIN FORMAT='VERBOSE' SELECT * FROM t1 WHERE ...; + + -- View the plan with the binding disabled + SET @@SESSION.TIDB_USE_PLAN_BASELINES = false; + EXPLAIN FORMAT='VERBOSE' SELECT * FROM t1 WHERE ...; + ``` + + - If the execution plan is consistent, you can delete the binding safely. + + - If the execution plan is inconsistent, you need to identify the cause, for examaple, checking operations on statistical information. In this case, you need retain the binding to ensure plan + ## Baseline evolution Baseline evolution is an important feature of SPM introduced in TiDB v4.0. @@ -271,7 +418,7 @@ Use the following statement to enable automatic binding evolution: {{< copyable "sql" >}} ```sql -set global tidb_evolve_plan_baselines = on; +SET GLOBAL tidb_evolve_plan_baselines = ON; ``` The default value of `tidb_evolve_plan_baselines` is `off`. @@ -288,7 +435,7 @@ Assume that table `t` is defined as follows: {{< copyable "sql" >}} ```sql -create table t(a int, b int, key(a), key(b)); +CREATE TABLE t(a INT, b INT, KEY(a), KEY(b)); ``` Perform the following query on table `t`: @@ -296,7 +443,7 @@ Perform the following query on table `t`: {{< copyable "sql" >}} ```sql -select * from t where a < 100 and b < 100; +SELECT * FROM t WHERE a < 100 AND b < 100; ``` In the table defined above, few rows meet the `a < 100` condition. But for some reason, the optimizer mistakenly selects the full table scan instead of the optimal execution plan that uses index `a`. You can first use the following statement to create a binding: @@ -304,7 +451,7 @@ In the table defined above, few rows meet the `a < 100` condition. But for some {{< copyable "sql" >}} ```sql -create global binding for select * from t where a < 100 and b < 100 using select * from t use index(a) where a < 100 and b < 100; +CREATE GLOBAL BINDING for SELECT * FROM t WHERE a < 100 AND b < 100 using SELECT * FROM t use index(a) WHERE a < 100 AND b < 100; ``` When the query above is executed again, the optimizer selects index `a` (influenced by the binding created above) to reduce the query time. @@ -343,18 +490,18 @@ Because the baseline evolution automatically creates a new binding, when the que During cluster upgrade, SQL Plan Management (SPM) might cause compatibility issues and make the upgrade fail. To ensure a successful upgrade, you need to include the following list for upgrade precheck: -* When you upgrade from a version earlier than v5.2.0 (that is, v4.0, v5.0, and v5.1) to the current version, make sure that `tidb_evolve_plan_baselines` is disabled before the upgrade. To disable this variable, perform the following steps. +* When you upgrade from a version earlier than v5.2.0 (that is, v4.0, v5.0, and v5.1) to the current version, make sure that `tidb_evolve_plan_baselines` is disabled before the upgrade. To disable this variable, perform the following steps. {{< copyable "sql" >}} ```sql - -- Check whether `tidb_evolve_plan_baselines` is disabled in the earlier version. - - select @@global.tidb_evolve_plan_baselines; - - -- If `tidb_evolve_plan_baselines` is still enabled, disable it. - - set global tidb_evolve_plan_baselines = off; + -- Check whether `tidb_evolve_plan_baselines` is disabled in the earlier version. + + SELECT @@global.tidb_evolve_plan_baselines; + + -- If `tidb_evolve_plan_baselines` is still enabled, disable it. + + SET GLOBAL tidb_evolve_plan_baselines = OFF; ``` * Before you upgrade from v4.0 to the current version, you need to check whether the syntax of all queries corresponding to the available SQL bindings is correct in the new version. If any syntax errors exist, delete the corresponding SQL binding. To do that, perform the following steps. @@ -363,15 +510,15 @@ During cluster upgrade, SQL Plan Management (SPM) might cause compatibility issu ```sql -- Check the query corresponding to the available SQL binding in the version to be upgraded. - - select bind_sql from mysql.bind_info where status = 'using'; - - -- Verify the result from the above SQL query in the test environment of the new version. - + + SELECT bind_sql FROM mysql.bind_info WHERE status = 'using'; + + -- Verify the result from the above SQL query in the test environment of the new version. + bind_sql_0; bind_sql_1; ... - - -- In the case of a syntax error (ERROR 1064 (42000): You have an error in your SQL syntax), delete the corresponding binding. - -- For any other errors (for example, tables are not found), it means that the syntax is compatible. No other operation is needed. + + -- In the case of a syntax error (ERROR 1064 (42000): You have an error in your SQL syntax), delete the corresponding binding. + -- For any other errors (for example, tables are not found), it means that the syntax is compatible. No other operation is needed. ``` diff --git a/system-variables.md b/system-variables.md index 0272b17d23595..43e8975964f24 100644 --- a/system-variables.md +++ b/system-variables.md @@ -1229,6 +1229,15 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - This variable is used to set the memory usage threshold of the local cache in the `Apply` operator. - The local cache in the `Apply` operator is used to speed up the computation of the `Apply` operator. You can set the variable to `0` to disable the `Apply` cache feature. +### `tidb_mem_quota_binding_cache` (New in v6.0) + +- Scope: GLOBAL +- Default value: `67108864` (64 MiB) +- Range: `[0, 2147483647]` +- Unit: Bytes +- This variable is used to set the threshold of the memory used for caching bindings. +- If a system creates or captures excessive bindings, resulting in overuse of memory space, TiDB returns a warning in the log. In this case, the cache cannot hold all available bindings or determine which bindings to store. For this reason, some queries may have no bindings. To address this problem, you can increase the value of this parameter, which will increase the memory space used for caching bindings. After modifying this parameter, you need to run `admin reload bindings` to reload bindings and validate the modification. + ### tidb_mem_quota_query - Scope: SESSION From 255d1f2184221448309b7518d8b20c59f5ffa800 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Tue, 29 Mar 2022 10:53:54 +0800 Subject: [PATCH 09/36] fix jenkins --- sql-plan-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 0956696c6b371..eac80b63ad802 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -357,7 +357,7 @@ Insert filtering conditions into the system table `mysql.capture_plan_baselines_ INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('user', 'user1'); ``` -| ** Dimension ** | ** Dimension name ** | ** Description ** | Remarks | +| ** Dimension** | ** Dimension name** | ** Description** | Remarks | | :----------- | :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | | Table name | table | Filter by table name. Each filtering rule is in the form of `db.table` and supports wildcards. For details about the rules, see [Plain table names](/table-filter.md# plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If the table name contains illegal content, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | | Frequency | frequency | SQL statements executed more than once are captured by default. You can set a high frequency to capture statements more frequently. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | From 4bc6d24b5eae2d83bc3336106ac7d28892d57181 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Tue, 29 Mar 2022 11:03:49 +0800 Subject: [PATCH 10/36] fix jenkins --- sql-plan-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index eac80b63ad802..67b1791b003eb 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -357,7 +357,7 @@ Insert filtering conditions into the system table `mysql.capture_plan_baselines_ INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('user', 'user1'); ``` -| ** Dimension** | ** Dimension name** | ** Description** | Remarks | +| **Dimension** | **Dimension name** | **Description** | Remarks | | :----------- | :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | | Table name | table | Filter by table name. Each filtering rule is in the form of `db.table` and supports wildcards. For details about the rules, see [Plain table names](/table-filter.md# plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If the table name contains illegal content, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | | Frequency | frequency | SQL statements executed more than once are captured by default. You can set a high frequency to capture statements more frequently. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | From 163b94a92c6e30b452a0520dae94d1c99970e13e Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Wed, 30 Mar 2022 17:32:02 +0800 Subject: [PATCH 11/36] Update sql-plan-management.md --- sql-plan-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 67b1791b003eb..7ec4d072e1130 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -326,7 +326,7 @@ However, TiDB does not automatically capture bindings for the following types of > **Note:** > -> Existing bindings are fixed by generating a group of hints. In this way, execution plans generated by query statements do not change. For most OLTP queries, including queries using the same index or Join mode (such as HashJoin and IndexJoin), TiDB guarantees plan consistency before and after binding. However, due to limitations of hints, TiDB cannot guarantee plan consistency for some complex queries, such as Join of more than two tables, MPP queries, and complex OLAP queries. +> Currently, a binding generates a group of hints to fix an execution plan generated by a query statement. In this way, the execution plan does not change. For most OLTP queries, including queries using the same index or Join algorithm (such as HashJoin and IndexJoin), TiDB guarantees plan consistency before and after the binding. However, due to the limitations of hints, TiDB cannot guarantee plan consistency for some complex queries, such as Join of more than two tables, MPP queries, and complex OLAP queries. For `PREPARE` / `EXECUTE` statements and for queries executed with binary protocols, TiDB automatically captures bindings for the real query statements, not for the `PREPARE` / `EXECUTE` statements. From fde9e7245a7860aaf15d83ff5eb702ea5bb158a1 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Thu, 31 Mar 2022 15:55:11 +0800 Subject: [PATCH 12/36] Apply suggestions from code review Co-authored-by: Chengpeng Yan <41809508+Reminiscent@users.noreply.github.com> --- sql-plan-management.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 7ec4d072e1130..d56d74a2e77fb 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -371,7 +371,7 @@ Insert filtering conditions into the system table `mysql.capture_plan_baselines_ ### Prevent rollback of execution plans during an upgrade -When upgrading a TiDB cluster, you can use baseline capturing to prevent rollback of execution plans by performing the following steps: +When upgrading a TiDB cluster, you can use baseline capturing to prevent regression of execution plans by performing the following steps: 1. Enable baseline capturing and keep it working for a period of time. This ensures that most important plans are captured. @@ -383,9 +383,9 @@ When upgrading a TiDB cluster, you can use baseline capturing to prevent rollbac 3. After the upgrade, delete bindings as required. - - Check the binding source by running the [`SHOW BINDINGS`](#view-binding) statement. + - Check the binding source by running the [`SHOW GLOBAL BINDINGS`](#view-binding) statement. - Based on the `Source` field in the output, decide whether the bindings are captured ( `capture`) or manually created (`manual`). + Based on the `Source` field in the output, decide whether the bindings are captured (`capture`) or manually created (`manual`). - Determine whether to retain the captured bindings: @@ -401,7 +401,7 @@ When upgrading a TiDB cluster, you can use baseline capturing to prevent rollbac - If the execution plan is consistent, you can delete the binding safely. - - If the execution plan is inconsistent, you need to identify the cause, for examaple, checking operations on statistical information. In this case, you need retain the binding to ensure plan + - If the execution plan is inconsistent, you need to identify the cause, for examaple, checking operations on statistical information. In this case, you need retain the binding to ensure plan consistency. ## Baseline evolution From 4b0ff94b382423981727f5254ea93173745c036c Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Thu, 31 Mar 2022 16:06:46 +0800 Subject: [PATCH 13/36] Address tech review comments --- sql-plan-management.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index d56d74a2e77fb..61565bb130b56 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -285,7 +285,7 @@ You can use either of the following methods to troubleshoot bindings: Each TiDB instace has a least recently used (LRU) cache for bindings. The cache capacity is controlled by the system variable [`tidb_mem_quota_binding_cache`](/system-variables.md#tidb_mem_quota_binding_cache-new-in-v60). You can view bindings that are cached in the TiDB instance. -To view the cache status of bindings, run the `SHOW binding_cache status` statement. In this statement, the effective scope is GLOBAL by default and cannot be modified. This statement returns the number of available bindings in the cache, the total number of available bindings in the system, memory usage of all cached bindings, and the memory quota. +To view the cache status of bindings, run the `SHOW binding_cache status` statement. In this statement, the effective scope is GLOBAL by default and cannot be modified. This statement returns the number of available bindings in the cache, the total number of available bindings in the system, memory usage of all cached bindings, and the total memory. {{< copyable "sql" >}} @@ -305,7 +305,7 @@ SHOW binding_cache status; ## Baseline capturing -Used for [preventing rollback of execution plans during an upgrade](#prevent-rollback-of-execution-plans-during-an-upgrade), this function captures queries that meet capturing conditions and create bindings for them. +Used for [preventing regression of execution plans during an upgrade](#prevent-regression-of-execution-plans-during-an-upgrade), this function captures queries that meet capturing conditions and create bindings for them. ### Enable capturing @@ -369,15 +369,15 @@ Insert filtering conditions into the system table `mysql.capture_plan_baselines_ > > - If the blocklist contains illegal filter content, TiDB returns warning message `[sql-bind] unknown capture filter type, ignore it` in the log. -### Prevent rollback of execution plans during an upgrade +### Prevent regression of execution plans during an upgrade When upgrading a TiDB cluster, you can use baseline capturing to prevent regression of execution plans by performing the following steps: -1. Enable baseline capturing and keep it working for a period of time. This ensures that most important plans are captured. +1. Enable baseline capturing and keep it working for a period of time. > **Note:** > - > Test data shows that long-term working of baseline capturing does not affect performance of the cluster load. Therefore, it is recommended to enable baseline capturing as long as possible. + > Test data shows that long-term working of baseline capturing does not affect performance of the cluster load. Therefore, it is recommended to enable baseline capturing as long as possible so that important plans are captured. 2. Upgrade the TiDB cluster. After the upgrade, captured bindings work to ensure plan consistency. From e69432f1c8aed8f487d69c949d96f1f6766180ad Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Thu, 31 Mar 2022 16:08:38 +0800 Subject: [PATCH 14/36] Update sql-plan-management.md Co-authored-by: Chengpeng Yan <41809508+Reminiscent@users.noreply.github.com> --- sql-plan-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 61565bb130b56..802c3586ab48b 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -24,7 +24,7 @@ This statement binds SQL execution plans at the GLOBAL or SESSION level. Current > **Note:** > -> Bindings have higher priority over hints added manually. Therefore, when you execute a statement containing a hint while a binding is present, the hint controlling the behavior of the optimizer does not take effect. However, other types of hints are still effective. +> Bindings have higher priority over manually added hints. Therefore, when you execute a statement containing a hint while a binding is present, the hint controlling the behavior of the optimizer does not take effect. However, other types of hints are still effective. Specifically, two types of these statements cannot be bound to execution plans due to syntax conflicts. See the following examples: From 26a6b81cccf811459293028699f027947fec1f9e Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Thu, 31 Mar 2022 16:10:46 +0800 Subject: [PATCH 15/36] Update sql-plan-management.md --- sql-plan-management.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 802c3586ab48b..85f14353267f7 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -202,7 +202,7 @@ You can execute this statement to change the status of a binding. The default st When executing this statement, you can only change the status of a binding from `Disabled` to `Enabled` or from `Enabled` to `Disabled`. If there is no binding for you to change the status, a warning message is returned, saying `There are no bindings can be set the status. Please check the SQL text`. Note that a binding in `Disabled` status is not used by any query. -### View binding +### View bindings {{< copyable "sql" >}} @@ -224,7 +224,7 @@ This statement outputs the execution plan bindings at the GLOBAL or SESSION leve | collation | Ordering rule | | source | The way in which a binding is created, including `manual` (created by the `create [global] binding` SQL statement), `capture` (captured automatically by TiDB), and `evolve` (evolved automatically by TiDB) | -### Troubleshoot bindings +### Troubleshoot a binding You can use either of the following methods to troubleshoot bindings: @@ -283,7 +283,7 @@ You can use either of the following methods to troubleshoot bindings: ### Cache bindings -Each TiDB instace has a least recently used (LRU) cache for bindings. The cache capacity is controlled by the system variable [`tidb_mem_quota_binding_cache`](/system-variables.md#tidb_mem_quota_binding_cache-new-in-v60). You can view bindings that are cached in the TiDB instance. +Each TiDB instance has a least recently used (LRU) cache for bindings. The cache capacity is controlled by the system variable [`tidb_mem_quota_binding_cache`](/system-variables.md#tidb_mem_quota_binding_cache-new-in-v60). You can view bindings that are cached in the TiDB instance. To view the cache status of bindings, run the `SHOW binding_cache status` statement. In this statement, the effective scope is GLOBAL by default and cannot be modified. This statement returns the number of available bindings in the cache, the total number of available bindings in the system, memory usage of all cached bindings, and the total memory. From 95394f8a07b1b194d5d7c8aac2dca61b67e4b6fd Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Thu, 31 Mar 2022 16:17:45 +0800 Subject: [PATCH 16/36] Update sql-plan-management.md Co-authored-by: Chengpeng Yan <41809508+Reminiscent@users.noreply.github.com> --- sql-plan-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 85f14353267f7..c4d87adb7bd54 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -226,7 +226,7 @@ This statement outputs the execution plan bindings at the GLOBAL or SESSION leve ### Troubleshoot a binding -You can use either of the following methods to troubleshoot bindings: +You can use either of the following methods to troubleshoot a binding: - Use the system variable [`last_plan_from_binding`](/system-variables.md#last_plan_from_binding-new-in-v40) to show whether the execution plan used by the last executed statement is from the binding. From cba9d7369697ebc9dc894f92e592cb5516b85cda Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Thu, 31 Mar 2022 16:37:07 +0800 Subject: [PATCH 17/36] Apply suggestions from code review Co-authored-by: Chengpeng Yan <41809508+Reminiscent@users.noreply.github.com> --- sql-plan-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index c4d87adb7bd54..881dfb63accb2 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -383,7 +383,7 @@ When upgrading a TiDB cluster, you can use baseline capturing to prevent regress 3. After the upgrade, delete bindings as required. - - Check the binding source by running the [`SHOW GLOBAL BINDINGS`](#view-binding) statement. + - Check the binding source by running the [`SHOW GLOBAL BINDINGS`](#view-bindings) statement. Based on the `Source` field in the output, decide whether the bindings are captured (`capture`) or manually created (`manual`). From 9a95ac4a197069f89f20e941e0134b2b3c015423 Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Thu, 31 Mar 2022 16:39:33 +0800 Subject: [PATCH 18/36] Simplify filter dimension table --- sql-plan-management.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 881dfb63accb2..0e92ea70955fb 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -357,11 +357,11 @@ Insert filtering conditions into the system table `mysql.capture_plan_baselines_ INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('user', 'user1'); ``` -| **Dimension** | **Dimension name** | **Description** | Remarks | -| :----------- | :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | -| Table name | table | Filter by table name. Each filtering rule is in the form of `db.table` and supports wildcards. For details about the rules, see [Plain table names](/table-filter.md# plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If the table name contains illegal content, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | -| Frequency | frequency | SQL statements executed more than once are captured by default. You can set a high frequency to capture statements more frequently. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | -| User name | user | Statements executed by blocklisted users are not captured. | If multiple users execute the same statement, this statement is not captured only if the user names are all in the blocklist. | +| **Dimension name** | **Description** | Remarks | +| :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | +| table | Filter by table name. Each filtering rule is in the form of `db.table` and supports wildcards. For details about the rules, see [Plain table names](/table-filter.md# plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If the table name contains illegal content, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | +| frequency | Filter by frequency. SQL statements executed more than once are captured by default. You can set a high frequency to capture statements more frequently. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | +| user | Filter by user name. Statements executed by blocklisted users are not captured. | If multiple users execute the same statement, this statement is not captured only if the user names are all in the blocklist. | > **Note:** > From 26eab07bb3c4d60531d07d8e9a6b156c9f45b5ba Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Thu, 31 Mar 2022 17:08:25 +0800 Subject: [PATCH 19/36] Update sql-plan-management.md --- sql-plan-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 0e92ea70955fb..24b7fd03094b4 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -377,7 +377,7 @@ When upgrading a TiDB cluster, you can use baseline capturing to prevent regress > **Note:** > - > Test data shows that long-term working of baseline capturing does not affect performance of the cluster load. Therefore, it is recommended to enable baseline capturing as long as possible so that important plans are captured. + > Test data shows that long-term working of baseline capturing does not affect performance of the cluster load. Therefore, it is recommended to enable baseline capturing as long as possible so that important plans (appear twice or above) are captured. 2. Upgrade the TiDB cluster. After the upgrade, captured bindings work to ensure plan consistency. From e71736c700dbad2e2ca8bc4dc6738632f962d8fd Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Thu, 31 Mar 2022 21:55:35 +0800 Subject: [PATCH 20/36] Apply suggestions from code review Co-authored-by: Grace Cai --- sql-plan-management.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 24b7fd03094b4..c0d4e7df98b68 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -24,7 +24,7 @@ This statement binds SQL execution plans at the GLOBAL or SESSION level. Current > **Note:** > -> Bindings have higher priority over manually added hints. Therefore, when you execute a statement containing a hint while a binding is present, the hint controlling the behavior of the optimizer does not take effect. However, other types of hints are still effective. +> Bindings have higher priority over manually added hints. Therefore, when you execute a statement containing a hint while a corresponding binding is present, the hint controlling the behavior of the optimizer does not take effect. However, other types of hints are still effective. Specifically, two types of these statements cannot be bound to execution plans due to syntax conflicts. See the following examples: @@ -198,9 +198,9 @@ In the example above, the dropped binding in the SESSION scope shields the corre SET BINDING [ENABLED | DISABLED] FOR BindableStmt; ``` -You can execute this statement to change the status of a binding. The default status is ENABLED. The effective scope is GLOBAL and cannot be modified. +You can execute this statement to change the status of a binding. The default status is ENABLED. The effective scope is GLOBAL by default and cannot be modified. -When executing this statement, you can only change the status of a binding from `Disabled` to `Enabled` or from `Enabled` to `Disabled`. If there is no binding for you to change the status, a warning message is returned, saying `There are no bindings can be set the status. Please check the SQL text`. Note that a binding in `Disabled` status is not used by any query. +When executing this statement, you can only change the status of a binding from `Disabled` to `Enabled` or from `Enabled` to `Disabled`. If no binding is available for status changes, a warning message is returned, saying `There are no bindings can be set the status. Please check the SQL text`. Note that a binding in `Disabled` status is not used by any query. ### View bindings @@ -217,7 +217,7 @@ This statement outputs the execution plan bindings at the GLOBAL or SESSION leve | original_sql | Original SQL statement after parameterization | | bind_sql | Bound SQL statement with hints | | default_db | Default database | -| status | Status including Enabled (available, replacing Using from v6.0), Disabled (unavailable), Deleted, Invalid, Rejected, and Pending verification| +| status | Status including Enabled (replacing the Using status from v6.0), Disabled, Deleted, Invalid, Rejected, and Pending verification| | create_time | Creating time | | update_time | Updating time | | charset | Character set | @@ -285,7 +285,7 @@ You can use either of the following methods to troubleshoot a binding: Each TiDB instance has a least recently used (LRU) cache for bindings. The cache capacity is controlled by the system variable [`tidb_mem_quota_binding_cache`](/system-variables.md#tidb_mem_quota_binding_cache-new-in-v60). You can view bindings that are cached in the TiDB instance. -To view the cache status of bindings, run the `SHOW binding_cache status` statement. In this statement, the effective scope is GLOBAL by default and cannot be modified. This statement returns the number of available bindings in the cache, the total number of available bindings in the system, memory usage of all cached bindings, and the total memory. +To view the cache status of bindings, run the `SHOW binding_cache status` statement. In this statement, the effective scope is GLOBAL by default and cannot be modified. This statement returns the number of available bindings in the cache, the total number of available bindings in the system, memory usage of all cached bindings, and the total memory for the cache. {{< copyable "sql" >}} @@ -305,7 +305,7 @@ SHOW binding_cache status; ## Baseline capturing -Used for [preventing regression of execution plans during an upgrade](#prevent-regression-of-execution-plans-during-an-upgrade), this function captures queries that meet capturing conditions and create bindings for them. +Used for [preventing regression of execution plans during an upgrade](#prevent-regression-of-execution-plans-during-an-upgrade), this feature captures queries that meet capturing conditions and creates bindings for these queries. ### Enable capturing @@ -322,11 +322,11 @@ However, TiDB does not automatically capture bindings for the following types of - `EXPLAIN` and `EXPLAIN ANALYZE` statements. - SQL statements executed internally in TiDB, such as `SELECT` queries used for automatically loading statistical information. - Statements that contain `Enabled` or `Disabled` bindings. -- Statements that meet capturing conditions. +- Statements that are filtered out by capturing conditions. > **Note:** > -> Currently, a binding generates a group of hints to fix an execution plan generated by a query statement. In this way, the execution plan does not change. For most OLTP queries, including queries using the same index or Join algorithm (such as HashJoin and IndexJoin), TiDB guarantees plan consistency before and after the binding. However, due to the limitations of hints, TiDB cannot guarantee plan consistency for some complex queries, such as Join of more than two tables, MPP queries, and complex OLAP queries. +> Currently, a binding generates a group of hints to fix an execution plan generated by a query statement. In this way, for the same query, the execution plan does not change. For most OLTP queries, including queries using the same index or Join algorithm (such as HashJoin and IndexJoin), TiDB guarantees plan consistency before and after the binding. However, due to the limitations of hints, TiDB cannot guarantee plan consistency for some complex queries, such as Join of more than two tables, MPP queries, and complex OLAP queries. For `PREPARE` / `EXECUTE` statements and for queries executed with binary protocols, TiDB automatically captures bindings for the real query statements, not for the `PREPARE` / `EXECUTE` statements. @@ -336,7 +336,7 @@ For `PREPARE` / `EXECUTE` statements and for queries executed with binary protoc ### Filter out bindings -This function allows you to configure a blocklist to filter out queries that meet the requirements. A blocklist has three dimensions, table name, frequency, and user name. +This feature allows you to configure a blocklist to filter out queries whose bindings you do not want to capture. A blocklist has three dimensions, table name, frequency, and user name. #### Usage From 25b170de775055519655298abb1e70e6d38e7ed8 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Thu, 31 Mar 2022 21:56:25 +0800 Subject: [PATCH 21/36] Update sql-plan-management.md --- sql-plan-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index c0d4e7df98b68..359e0361e2faf 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -401,7 +401,7 @@ When upgrading a TiDB cluster, you can use baseline capturing to prevent regress - If the execution plan is consistent, you can delete the binding safely. - - If the execution plan is inconsistent, you need to identify the cause, for examaple, checking operations on statistical information. In this case, you need retain the binding to ensure plan consistency. + - If the execution plan is inconsistent, you need to identify the cause, for examaple, checking operations on statistical information. In this case, you need to retain the binding to ensure plan consistency. ## Baseline evolution From 29d00b7bc4b257bdd4664d2daf4037e9a86b10c9 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Fri, 1 Apr 2022 11:48:13 +0800 Subject: [PATCH 22/36] Apply suggestions from code review Co-authored-by: Grace Cai --- sql-plan-management.md | 12 ++++++------ system-variables.md | 4 ++-- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 359e0361e2faf..da2f58b4bf7c6 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -359,7 +359,7 @@ Insert filtering conditions into the system table `mysql.capture_plan_baselines_ | **Dimension name** | **Description** | Remarks | | :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | -| table | Filter by table name. Each filtering rule is in the form of `db.table` and supports wildcards. For details about the rules, see [Plain table names](/table-filter.md# plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If the table name contains illegal content, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | +| table | Filter by table name. Each filtering rule is in the `db.table` format. The supported filtering syntax includes [Plain table names](/table-filter.md#plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If a table name contains illegal characters, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | | frequency | Filter by frequency. SQL statements executed more than once are captured by default. You can set a high frequency to capture statements more frequently. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | | user | Filter by user name. Statements executed by blocklisted users are not captured. | If multiple users execute the same statement, this statement is not captured only if the user names are all in the blocklist. | @@ -367,11 +367,11 @@ Insert filtering conditions into the system table `mysql.capture_plan_baselines_ > > - Modifying a blocklist requires the super privilege. > -> - If the blocklist contains illegal filter content, TiDB returns warning message `[sql-bind] unknown capture filter type, ignore it` in the log. +> - If a blocklist contains illegal filters, TiDB returns the warning message `[sql-bind] unknown capture filter type, ignore it` in the log. ### Prevent regression of execution plans during an upgrade -When upgrading a TiDB cluster, you can use baseline capturing to prevent regression of execution plans by performing the following steps: + Before upgrading a TiDB cluster, you can use baseline capturing to prevent regression of execution plans by performing the following steps: 1. Enable baseline capturing and keep it working for a period of time. @@ -379,13 +379,13 @@ When upgrading a TiDB cluster, you can use baseline capturing to prevent regress > > Test data shows that long-term working of baseline capturing does not affect performance of the cluster load. Therefore, it is recommended to enable baseline capturing as long as possible so that important plans (appear twice or above) are captured. -2. Upgrade the TiDB cluster. After the upgrade, captured bindings work to ensure plan consistency. +2. Upgrade the TiDB cluster. After the upgrade, TiDB uses those captured bindings to ensure execution plan consistency. 3. After the upgrade, delete bindings as required. - Check the binding source by running the [`SHOW GLOBAL BINDINGS`](#view-bindings) statement. - Based on the `Source` field in the output, decide whether the bindings are captured (`capture`) or manually created (`manual`). + In the output, check the `Source` field to see whether a binding is captured (`capture`) or manually created (`manual`). - Determine whether to retain the captured bindings: @@ -401,7 +401,7 @@ When upgrading a TiDB cluster, you can use baseline capturing to prevent regress - If the execution plan is consistent, you can delete the binding safely. - - If the execution plan is inconsistent, you need to identify the cause, for examaple, checking operations on statistical information. In this case, you need to retain the binding to ensure plan consistency. + - If the execution plan is inconsistent, you need to identify the cause, for example, by checking statistics. In this case, you need to retain the binding to ensure plan consistency. ## Baseline evolution diff --git a/system-variables.md b/system-variables.md index 43e8975964f24..dea308d39c802 100644 --- a/system-variables.md +++ b/system-variables.md @@ -1229,14 +1229,14 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - This variable is used to set the memory usage threshold of the local cache in the `Apply` operator. - The local cache in the `Apply` operator is used to speed up the computation of the `Apply` operator. You can set the variable to `0` to disable the `Apply` cache feature. -### `tidb_mem_quota_binding_cache` (New in v6.0) +### tidb_mem_quota_binding_cache (New in v6.0.0) - Scope: GLOBAL - Default value: `67108864` (64 MiB) - Range: `[0, 2147483647]` - Unit: Bytes - This variable is used to set the threshold of the memory used for caching bindings. -- If a system creates or captures excessive bindings, resulting in overuse of memory space, TiDB returns a warning in the log. In this case, the cache cannot hold all available bindings or determine which bindings to store. For this reason, some queries may have no bindings. To address this problem, you can increase the value of this parameter, which will increase the memory space used for caching bindings. After modifying this parameter, you need to run `admin reload bindings` to reload bindings and validate the modification. +- If a system creates or captures excessive bindings, resulting in overuse of memory space, TiDB returns a warning in the log. In this case, the cache cannot hold all available bindings or determine which bindings to store. For this reason, some queries might miss their bindings. To address this problem, you can increase the value of this variable, which increases the memory used for caching bindings. After modifying this parameter, you need to run `admin reload bindings` to reload bindings and validate the modification. ### tidb_mem_quota_query From 0f45a148288500d44becc266a260fcb1d46d2e8d Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Fri, 1 Apr 2022 17:12:16 +0800 Subject: [PATCH 23/36] Apply suggestions from code review --- sql-plan-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index da2f58b4bf7c6..2edc611e03b8b 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -361,7 +361,7 @@ Insert filtering conditions into the system table `mysql.capture_plan_baselines_ | :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | | table | Filter by table name. Each filtering rule is in the `db.table` format. The supported filtering syntax includes [Plain table names](/table-filter.md#plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If a table name contains illegal characters, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | | frequency | Filter by frequency. SQL statements executed more than once are captured by default. You can set a high frequency to capture statements more frequently. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | -| user | Filter by user name. Statements executed by blocklisted users are not captured. | If multiple users execute the same statement, this statement is not captured only if the user names are all in the blocklist. | +| user | Filter by user name. Statements executed by blocklisted users are not captured. | If multiple users execute the same statement and their user names are all in the blocklist, this statement is captured. | > **Note:** > From dcc6fa365d701138716e2f5e7deeb041c8eaf2ad Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Fri, 1 Apr 2022 17:22:53 +0800 Subject: [PATCH 24/36] Apply suggestions from code review --- sql-plan-management.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 2edc611e03b8b..a6da9d76a9c33 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -360,7 +360,7 @@ Insert filtering conditions into the system table `mysql.capture_plan_baselines_ | **Dimension name** | **Description** | Remarks | | :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | | table | Filter by table name. Each filtering rule is in the `db.table` format. The supported filtering syntax includes [Plain table names](/table-filter.md#plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If a table name contains illegal characters, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | -| frequency | Filter by frequency. SQL statements executed more than once are captured by default. You can set a high frequency to capture statements more frequently. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | +| frequency | Filter by frequency. SQL statements executed more than once are captured by default. You can set a high frequency to capture statements that are frequently executed. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | | user | Filter by user name. Statements executed by blocklisted users are not captured. | If multiple users execute the same statement and their user names are all in the blocklist, this statement is captured. | > **Note:** @@ -377,7 +377,7 @@ Insert filtering conditions into the system table `mysql.capture_plan_baselines_ > **Note:** > - > Test data shows that long-term working of baseline capturing does not affect performance of the cluster load. Therefore, it is recommended to enable baseline capturing as long as possible so that important plans (appear twice or above) are captured. + > Test data shows that long-term working of baseline capturing has a slight impact on the performance of the cluster load. Therefore, it is recommended to enable baseline capturing as long as possible so that important plans (appear twice or above) are captured. 2. Upgrade the TiDB cluster. After the upgrade, TiDB uses those captured bindings to ensure execution plan consistency. From 1314827afdd50ed567d2aaf8685e9596f1ed2612 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Fri, 1 Apr 2022 17:29:27 +0800 Subject: [PATCH 25/36] change v60 to v600 --- sql-plan-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index a6da9d76a9c33..eb018db954b3f 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -283,7 +283,7 @@ You can use either of the following methods to troubleshoot a binding: ### Cache bindings -Each TiDB instance has a least recently used (LRU) cache for bindings. The cache capacity is controlled by the system variable [`tidb_mem_quota_binding_cache`](/system-variables.md#tidb_mem_quota_binding_cache-new-in-v60). You can view bindings that are cached in the TiDB instance. +Each TiDB instance has a least recently used (LRU) cache for bindings. The cache capacity is controlled by the system variable [`tidb_mem_quota_binding_cache`](/system-variables.md#tidb_mem_quota_binding_cache-new-in-v600). You can view bindings that are cached in the TiDB instance. To view the cache status of bindings, run the `SHOW binding_cache status` statement. In this statement, the effective scope is GLOBAL by default and cannot be modified. This statement returns the number of available bindings in the cache, the total number of available bindings in the system, memory usage of all cached bindings, and the total memory for the cache. From 0fe95644f2918403d4fe63713790d1b58a109391 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Fri, 1 Apr 2022 19:44:46 +0800 Subject: [PATCH 26/36] Update sql-plan-management.md --- sql-plan-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index eb018db954b3f..8f7be10ccd162 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -361,7 +361,7 @@ Insert filtering conditions into the system table `mysql.capture_plan_baselines_ | :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | | table | Filter by table name. Each filtering rule is in the `db.table` format. The supported filtering syntax includes [Plain table names](/table-filter.md#plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If a table name contains illegal characters, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | | frequency | Filter by frequency. SQL statements executed more than once are captured by default. You can set a high frequency to capture statements that are frequently executed. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | -| user | Filter by user name. Statements executed by blocklisted users are not captured. | If multiple users execute the same statement and their user names are all in the blocklist, this statement is captured. | +| user | Filter by user name. Statements executed by blocklisted users are not captured. | If multiple users execute the same statement and their user names are all in the blocklist, this statement is not captured. | > **Note:** > From 818257050ef9e7b58318c2e1eb7a3bd026ce5976 Mon Sep 17 00:00:00 2001 From: Enwei Date: Fri, 1 Apr 2022 21:12:29 +0800 Subject: [PATCH 27/36] TiDB Scheduling: Revise the format of status description parts (#8004) --- tidb-scheduling.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/tidb-scheduling.md b/tidb-scheduling.md index 564bb2d3247f3..4beb76c372467 100644 --- a/tidb-scheduling.md +++ b/tidb-scheduling.md @@ -77,15 +77,15 @@ Scheduling is based on information collection. In short, the PD scheduling compo * Whether the store is overloaded * Labels (See [Perception of Topology](/schedule-replicas-by-topology-labels.md)) -You can use PD control to check the status of a TiKV store, which can be Up, Disconnect, Offline, Down, or Tombstone. The following is a description of all statuses and their relationship. + You can use PD control to check the status of a TiKV store, which can be Up, Disconnect, Offline, Down, or Tombstone. The following is a description of all statuses and their relationship. -+ **Up**: The TiKV store is in service. -+ **Disconnect**: Heartbeat messages between the PD and the TiKV store are lost for more than 20 seconds. If the lost period exceeds the time specified by `max-store-down-time`, the status "Disconnect" changes to "Down". -+ **Down**: Heartbeat messages between the PD and the TiKV store are lost for a time longer than `max-store-down-time` (30 minutes by default). In this status, the TiKV store starts replenishing replicas of each Region on the surviving store. -+ **Offline**: A TiKV store is manually taken offline through PD Control. This is only an intermediate status for the store to go offline. The store in this status moves all its Regions to other "Up" stores that meet the relocation conditions. When `leader_count` and `region_count` (obtained through PD Control) both show `0`, the store status changes to "Tombstone" from "Offline". In the "Offline" status, **do not** disable the store service or the physical server where the store is located. During the process that the store goes offline, if the cluster does not have target stores to relocate the Regions (for example, inadequate stores to hold replicas in the cluster), the store is always in the "Offline" status. -+ **Tombstone**: The TiKV store is completely offline. You can use the `remove-tombstone` interface to safely clean up TiKV in this status. + + **Up**: The TiKV store is in service. + + **Disconnect**: Heartbeat messages between the PD and the TiKV store are lost for more than 20 seconds. If the lost period exceeds the time specified by `max-store-down-time`, the status "Disconnect" changes to "Down". + + **Down**: Heartbeat messages between the PD and the TiKV store are lost for a time longer than `max-store-down-time` (30 minutes by default). In this status, the TiKV store starts replenishing replicas of each Region on the surviving store. + + **Offline**: A TiKV store is manually taken offline through PD Control. This is only an intermediate status for the store to go offline. The store in this status moves all its Regions to other "Up" stores that meet the relocation conditions. When `leader_count` and `region_count` (obtained through PD Control) both show `0`, the store status changes to "Tombstone" from "Offline". In the "Offline" status, **do not** disable the store service or the physical server where the store is located. During the process that the store goes offline, if the cluster does not have target stores to relocate the Regions (for example, inadequate stores to hold replicas in the cluster), the store is always in the "Offline" status. + + **Tombstone**: The TiKV store is completely offline. You can use the `remove-tombstone` interface to safely clean up TiKV in this status. -![TiKV store status relationship](/media/tikv-store-status-relationship.png) + ![TiKV store status relationship](/media/tikv-store-status-relationship.png) - Information reported by Region leaders: From e84dc58f0b1c475d4f9f9fce40f5df94835a6d99 Mon Sep 17 00:00:00 2001 From: Enwei Date: Fri, 1 Apr 2022 21:14:30 +0800 Subject: [PATCH 28/36] BR: Add a new doc about the batch create table (#7983) --- TOC.md | 2 ++ br/backup-and-restore-faq.md | 6 ++++ br/br-batch-create-table.md | 65 ++++++++++++++++++++++++++++++++++++ 3 files changed, 73 insertions(+) create mode 100644 br/br-batch-create-table.md diff --git a/TOC.md b/TOC.md index b662aa456ca4b..83dccb5bcccbe 100644 --- a/TOC.md +++ b/TOC.md @@ -72,6 +72,7 @@ - [Back up and Restore Data on Azure Blob Storage](/br/backup-and-restore-azblob.md) - BR Features - [Auto Tune](/br/br-auto-tune.md) + - [Batch Create Table](/br/br-batch-create-table.md) - [BR FAQ](/br/backup-and-restore-faq.md) - [Configure Time Zone](/configure-time-zone.md) - [Daily Checklist](/daily-check.md) @@ -203,6 +204,7 @@ - [External Storages](/br/backup-and-restore-storages.md) - BR Features - [Auto Tune](/br/br-auto-tune.md) + - [Batch Create Table](/br/br-batch-create-table.md) - [BR FAQ](/br/backup-and-restore-faq.md) - TiDB Binlog - [Overview](/tidb-binlog/tidb-binlog-overview.md) diff --git a/br/backup-and-restore-faq.md b/br/backup-and-restore-faq.md index 6dca31b9c41c4..2e6ece4643fc2 100644 --- a/br/backup-and-restore-faq.md +++ b/br/backup-and-restore-faq.md @@ -164,6 +164,12 @@ You can use [`filter.rules`](https://github.com/pingcap/tiflow/blob/7c3c2336f981 Yes. BR backs up the [`SHARD_ROW_ID_BITS` and `PRE_SPLIT_REGIONS`](/sql-statements/sql-statement-split-region.md#pre_split_regions) information of a table. The data of the restored table is also split into multiple Regions. +## What should I do if the restore fails with the error message `the entry too large, the max entry size is 6291456, the size of data is 7690800`? + +You can try to reduce the number of tables to be created in a batch by setting `--ddl-batch-size` to `128` or a smaller value. + +When using BR to restore the backup data with the value of [`--ddl-batch-size`](/br/br-batch-create-table.md#how to use) greater than `1`, TiDB writes a DDL job of table creation to the DDL jobs queue that is maintained by TiKV. At this time, the total size of all tables schema sent by TiDB at one time should not exceed 6 MB, because the maximum value of job messages is `6 MB` by default (it is **not recommended** to modify this value. For details, see [`txn-entry-size-limit`](/tidb-configuration-file.md#txn-entry-size-limit-new-in-v50) and [`raft-entry-max-size`](/tikv-configuration-file.md#raft-entry-max-size)). Therefore, if you set `--ddl-batch-size` to an excessively large value, the schema size of the tables sent by TiDB in a batch at one time exceeds the specified value, which causes BR to report the `entry too large, the max entry size is 6291456, the size of data is 7690800` error. + ## Why is the `region is unavailable` error reported for a SQL query after I use BR to restore the backup data? If the cluster backed up using BR has TiFlash, `TableInfo` stores the TiFlash information when BR restores the backup data. If the cluster to be restored does not have TiFlash, the `region is unavailable` error is reported. diff --git a/br/br-batch-create-table.md b/br/br-batch-create-table.md new file mode 100644 index 0000000000000..b10e066f6185c --- /dev/null +++ b/br/br-batch-create-table.md @@ -0,0 +1,65 @@ +--- +title: Batch Create Table +summary: Learn how to use the Batch Create Table feature. When restoring data, BR can create tables in batches to speed up the restore process. +--- + +# Batch Create Table + +When restoring data, Backup & Restore (BR) creates databases and tables in the target TiDB cluster and then restores the backup data to the tables. In versions earlier than TiDB v6.0.0, BR uses the [serial execution](#implementation-principles) implementation to create tables in the restore process. However, when BR restores data with a large number of tables (nearly 50000), this implementation takes much time on creating tables. + +To speed up the table creation process and reduce the time for restoring data, the Batch Create Table feature is introduced in TiDB v6.0.0. This feature is enabled by default. + +> **Note:** +> +> - To use the Batch Create Table feature, both TiDB and BR are expected to be of v6.0.0 or later. If either TiDB or BR is earlier than v6.0.0, BR uses the serial execution implementation. +> - Suppose that you use a cluster management tool (for example, TiUP), and your TiDB and BR are of v6.0.0 or later versions, or your TiDB and BR are upgraded from a version earlier than v6.0.0 to v6.0.0 or later. In this case, BR enables the Batch Create Table feature by default. + +## Usage scenario + +If you need to restore data with a massive amount of tables, for example, 50000 tables, you can use the Batch Create Table feature to speed up the restore process. + +For the detailed effect, see [Test for the Batch Create Table Feature](#test-for-the-batch-create-table-feature). + +## Use the Batch Create Table feature + +BR enables the Batch Create Table feature by default, with the default configuration of `--ddl-batch-size=128` in v6.0.0 or later to speed up the restore process. Therefore, you do not need to configure this parameter. `--ddl-batch-size=128` means that BR creates tables in batches, each batch with 128 tables. + +To disable this feature, you can set `--ddl-batch-size` to `0`. See the following example command: + +{{< copyable "shell-regular" >}} + +```shell +br restore full -s local:///br_data/ --pd 172.16.5.198:2379 --log-file restore.log --ddl-batch-size=0 +``` + +After this feature is disabled, BR uses the [serial execution implementation](#implementation-principles) instead. + +## Implementation principles + +- Serial execution implementation before v6.0.0: + + When restoring data, BR creates databases and tables in the target TiDB cluster and then restores the backup data to the tables. To create tables, BR calls TiDB internal API first, and then processes table creation tasks, which works similarly to executing the `Create Table` statement by BR. The TiDB DDL owner creates tables sequentially. Once the DDL owner creates a table, the DDL schema version changes correspondingly and each version change is synchronized to other TiDB DDL workers (including BR). Therefore, when BR restores a large number of tables, the serial execution implementation is time-consuming. + +- Batch create table implementation since v6.0.0: + + By default, BR creates tables in multiple batches, and each batch has 128 tables. Using this implementation, when BR creates one batch of tables, the TiDB schema version only changes once. This implementation significantly increases the speed of table creation. + +## Test for the Batch Create Table feature + +This section describes the test information about the Batch Create Table feature. The test environment is as follows: + +- Cluster configurations: + + - 15 TiKV instances. Each TiKV instance is equipped with 16 CPU cores, 80 GB memory, and 16 threads to process RPC requests ([`import.num-threads`](/tikv-configuration-file.md#num-threads) = 16). + - 3 TiDB instances. Each TiDB instance is equipped with 16 CPU cores, 32 GB memory. + - 3 PD instances. Each PD instance is equipped with 16 CPU cores, 32 GB memory. + +- The size of data to be restored: 16.16 TB + +The test result is as follows: + +``` +‘[2022/03/12 22:37:49.060 +08:00] [INFO] [collector.go:67] ["Full restore success summary"] [total-ranges=751760] [ranges-succeed=751760] [ranges-failed=0] [split-region=1h33m18.078448449s] [restore-ranges=542693] [total-take=1h41m35.471476438s] [restore-data-size(after-compressed)=8.337TB] [Size=8336694965072] [BackupTS=431773933856882690] [total-kv=148015861383] [total-kv-size=16.16TB] [average-speed=2.661GB/s]’ +``` + +From the test result, you can see that the average speed of restoring one TiKV instance is as high as 181.65 MB/s (which equals to `average-speed`/`tikv_count`）. \ No newline at end of file From e7df687bffd13994e818bb95cdf9f703075f708c Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Sat, 2 Apr 2022 10:16:30 +0800 Subject: [PATCH 29/36] add doc for new-collation-enable in br (#8031) --- br/backup-and-restore-tool.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 4ec4e973f1d67..1892bdb221269 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -156,10 +156,21 @@ Note that skipping the version check might introduce incompatibility. The versio | Backup version (vertical) \ Restore version (horizontal) | Use BR nightly to restore TiDB nightly | Use BR v5.0 to restore TiDB v5.0| Use BR v4.0 to restore TiDB v4.0 | | ---- | ---- | ---- | ---- | | Use BR nightly to back up TiDB nightly | ✅ | ✅ | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | -| Use BR v5.0 to back up TiDB v5.0 | ✅ | ✅ | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) +| Use BR v5.0 to back up TiDB v5.0 | ✅ | ✅ | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | Use BR v4.0 to back up TiDB v4.0 | ✅ | ✅ | ✅ (If TiKV >= v4.0.0-rc.1, and if BR contains the [#233](https://github.com/pingcap/br/pull/233) bug fix and TiKV does not contain the [#7241](https://github.com/tikv/tikv/pull/7241) bug fix, BR will cause the TiKV node to restart.) | | Use BR nightly or v5.0 to back up TiDB v4.0 | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | +#### Check for the `new_collations_enabled_on_first_bootstrap` variable + +Since TiDB v6.0.0, the default value of [`new_collations_enabled_on_first_bootstrap`](/tidb-configuration-file.md#new_collations_enabled_on_first_bootstrap) has changed from `false` to `true`. When the value of `new_collations_enabled_on_first_bootstrap` is consistent between the upstream and downstream clusters, BR safely restores the data backed up in the upstream cluster to the downstream cluster. + +Since v6.0.0, BR backs up the `new_collations_enabled_on_first_bootstrap` configuration of the upstream cluster and then checks whether the value of this configuration is consistent between the upstream and downstream clusters. If the value is inconsistent between the upstream and downstream clusters, BR does not perform the data restore and reports an error. + +Suppose that you have backed up the data in a TiDB cluster of an earlier version of v6.0.0, and you want to restore this data to a TiDB cluster of v6.0.0 or later versions. In this situation, you need manually to check whether the value of `new_collations_enabled_on_first_bootstrap` is consistent between the upstream and downstream clusters: + +- If the value is consistent, you can add `--check-requirements=false` to the restore command to skip this configuration check. +- If the value is inconsistent, and you forcibly perform the restore, BR reports a [data validation error](/br/backup-and-restore-tool.md#usage-restrictions). + ### Back up and restore table data in the `mysql` system schema (experimental feature) > **Warning:** From 361dee2e5a6d6fb1246683f57f2e4e9371f9f5a4 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Sat, 2 Apr 2022 11:54:30 +0800 Subject: [PATCH 30/36] update the release info of TiDB Cloud (#7986) --- _index.md | 2 +- overview.md | 2 +- quick-start-with-tidb.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/_index.md b/_index.md index ac79c1dc3a156..55d9035631407 100644 --- a/_index.md +++ b/_index.md @@ -8,7 +8,7 @@ aliases: ['/docs/dev/'] [TiDB](https://github.com/pingcap/tidb) (/’taɪdiːbi:/, "Ti" stands for Titanium) is an open-source, distributed, NewSQL database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads. It is MySQL compatible and features horizontal scalability, strong consistency, and high availability. TiDB can be deployed on-premise or in-cloud. -Designed for the cloud, TiDB provides flexible scalability, reliability and security on the cloud platform. Users can elastically scale TiDB to meet the requirements of their changing workloads. [TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/v1.1/tidb-operator-overview) helps manage TiDB on Kubernetes and automates operating tasks, which makes TiDB easier to deploy on any cloud that provides managed Kubernetes. [TiDB Cloud](https://pingcap.com/products/tidbcloud) (Public Preview), the fully-managed TiDB service, is the easiest, most economical, and most resilient way to unlock the full power of [TiDB in the cloud](https://docs.pingcap.com/tidbcloud/beta), allowing you to deploy and run TiDB clusters with just a few clicks. +Designed for the cloud, TiDB provides flexible scalability, reliability and security on the cloud platform. Users can elastically scale TiDB to meet the requirements of their changing workloads. [TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/v1.1/tidb-operator-overview) helps manage TiDB on Kubernetes and automates operating tasks, which makes TiDB easier to deploy on any cloud that provides managed Kubernetes. [TiDB Cloud](https://pingcap.com/tidb-cloud/), the fully-managed TiDB service, is the easiest, most economical, and most resilient way to unlock the full power of [TiDB in the cloud](https://docs.pingcap.com/tidbcloud/), allowing you to deploy and run TiDB clusters with just a few clicks. diff --git a/overview.md b/overview.md index 772d6fdd6918e..c9532807c16fd 100644 --- a/overview.md +++ b/overview.md @@ -26,7 +26,7 @@ aliases: ['/docs/dev/key-features/','/tidb/dev/key-features','/docs/dev/overview - **Cloud-native distributed database** - TiDB is a distributed database designed for the cloud, providing flexible scalability, reliability and security on the cloud platform. Users can elastically scale TiDB to meet the requirements of their changing workloads. In TiDB, each piece of data has 3 replicas at least, which can be scheduled in different cloud availability zones to tolerate the outage of a whole data center. [TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/stable/tidb-operator-overview) helps manage TiDB on Kubernetes and automates tasks related to operating the TiDB cluster, which makes TiDB easier to deploy on any cloud that provides managed Kubernetes. [TiDB Cloud](https://pingcap.com/products/tidbcloud) (Beta), the fully-managed TiDB service, is the easiest, most economical, and most resilient way to unlock the full power of [TiDB in the cloud](https://docs.pingcap.com/tidbcloud/beta), allowing you to deploy and run TiDB clusters with just a few clicks. + TiDB is a distributed database designed for the cloud, providing flexible scalability, reliability and security on the cloud platform. Users can elastically scale TiDB to meet the requirements of their changing workloads. In TiDB, each piece of data has 3 replicas at least, which can be scheduled in different cloud availability zones to tolerate the outage of a whole data center. [TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/stable/tidb-operator-overview) helps manage TiDB on Kubernetes and automates tasks related to operating the TiDB cluster, which makes TiDB easier to deploy on any cloud that provides managed Kubernetes. [TiDB Cloud](https://pingcap.com/tidb-cloud/), the fully-managed TiDB service, is the easiest, most economical, and most resilient way to unlock the full power of [TiDB in the cloud](https://docs.pingcap.com/tidbcloud/), allowing you to deploy and run TiDB clusters with just a few clicks. - **Compatible with the MySQL 5.7 protocol and MySQL ecosystem** diff --git a/quick-start-with-tidb.md b/quick-start-with-tidb.md index e00f8ae14fcb2..7d090bc96fc03 100644 --- a/quick-start-with-tidb.md +++ b/quick-start-with-tidb.md @@ -19,7 +19,7 @@ This guide walks you through the quickest way to get started with TiDB. For non- > > - To deploy an on-premises production cluster, see [production installation guide](/production-deployment-using-tiup.md). > - To deploy TiDB in Kubernetes, see [Get Started with TiDB in Kubernetes](https://docs.pingcap.com/tidb-in-kubernetes/stable/get-started). -> - To manage TiDB in the cloud, see [TiDB Cloud Quick Start](https://docs.pingcap.com/tidbcloud/beta/tidb-cloud-quickstart). +> - To manage TiDB in the cloud, see [TiDB Cloud Quick Start](https://docs.pingcap.com/tidbcloud/tidb-cloud-quickstart). ## Deploy a local test cluster From 795edae8832aba4dbe6e8bf18bcebbb6a4e4d0d1 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Sat, 2 Apr 2022 11:56:29 +0800 Subject: [PATCH 31/36] tiup bump version for 6.0 (#8038) --- production-deployment-using-tiup.md | 12 ++++----- quick-start-with-tidb.md | 8 +++--- scale-tidb-using-tiup.md | 4 +-- tiup/tiup-bench.md | 1 - tiup/tiup-cluster.md | 36 +++++++++++++-------------- tiup/tiup-component-cluster-deploy.md | 2 +- tiup/tiup-component-management.md | 12 ++++----- tiup/tiup-mirror.md | 6 ++--- tiup/tiup-playground.md | 4 +-- upgrade-tidb-using-tiup.md | 35 ++++++++++++-------------- 10 files changed, 58 insertions(+), 62 deletions(-) diff --git a/production-deployment-using-tiup.md b/production-deployment-using-tiup.md index 9836b27743729..90c1adfc200d4 100644 --- a/production-deployment-using-tiup.md +++ b/production-deployment-using-tiup.md @@ -141,12 +141,12 @@ To prepare the TiUP offline component package, manually pack an offline componen If you want to adjust an existing offline mirror (such as adding a new version of a component), take the following steps: - 1. When pulling an offline mirror, you can get an incomplete offline mirror by specifying specific information via parameters, such as the component and version information. For example, you can pull an offline mirror that includes only the offline mirror of TiUP v1.9.0 and TiUP Cluster v1.9.0 by running the following command: + 1. When pulling an offline mirror, you can get an incomplete offline mirror by specifying specific information via parameters, such as the component and version information. For example, you can pull an offline mirror that includes only the offline mirror of TiUP v1.9.3 and TiUP Cluster v1.9.3 by running the following command: {{< copyable "shell-regular" >}} ```bash - tiup mirror clone tiup-custom-mirror-v1.9.0 --tiup v1.9.0 --cluster v1.9.0 + tiup mirror clone tiup-custom-mirror-v1.9.3 --tiup v1.9.3 --cluster v1.9.3 ``` If you only need the components for a particular platform, you can specify them using the `--os` or `--arch` parameters. @@ -178,10 +178,10 @@ To prepare the TiUP offline component package, manually pack an offline componen {{< copyable "shell-regular" >}} ```bash - tiup mirror merge tiup-custom-mirror-v1.9.0 + tiup mirror merge tiup-custom-mirror-v1.9.3 ``` - 5. When the above steps are completed, check the result by running the `tiup list` command. In this document's example, the outputs of both `tiup list tiup` and `tiup list cluster` show that the corresponding components of `v1.9.0` are available. + 5. When the above steps are completed, check the result by running the `tiup list` command. In this document's example, the outputs of both `tiup list tiup` and `tiup list cluster` show that the corresponding components of `v1.9.3` are available. #### Step 2: Deploy the offline TiUP component @@ -310,13 +310,13 @@ Then execute the `deploy` command to deploy the TiDB cluster: {{< copyable "shell-regular" >}} ```shell -tiup cluster deploy tidb-test v5.4.0 ./topology.yaml --user root [-p] [-i /home/root/.ssh/gcp_rsa] +tiup cluster deploy tidb-test v6.0.0 ./topology.yaml --user root [-p] [-i /home/root/.ssh/gcp_rsa] ``` In the above command: - The name of the deployed TiDB cluster is `tidb-test`. -- You can see the latest supported versions by running `tiup list tidb`. This document takes `v5.4.0` as an example. +- You can see the latest supported versions by running `tiup list tidb`. This document takes `v6.0.0` as an example. - The initialization configuration file is `topology.yaml`. - `--user root`: Log in to the target machine through the `root` key to complete the cluster deployment, or you can use other users with `ssh` and `sudo` privileges to complete the deployment. - `[-i]` and `[-p]`: optional. If you have configured login to the target machine without password, these parameters are not required. If not, choose one of the two parameters. `[-i]` is the private key of the `root` user (or other users specified by `--user`) that has access to the target machine. `[-p]` is used to input the user password interactively. diff --git a/quick-start-with-tidb.md b/quick-start-with-tidb.md index 7d090bc96fc03..d5b27dd689662 100644 --- a/quick-start-with-tidb.md +++ b/quick-start-with-tidb.md @@ -65,10 +65,10 @@ As a distributed system, a basic TiDB test cluster usually consists of 2 TiDB in {{< copyable "shell-regular" >}} ```shell - tiup playground v5.4.0 --db 2 --pd 3 --kv 3 + tiup playground v6.0.0 --db 2 --pd 3 --kv 3 ``` - The command downloads a version cluster to the local machine and starts it, such as v5.4.0. To view the latest version, run `tiup list tidb`. + The command downloads a version cluster to the local machine and starts it, such as v6.0.0. To view the latest version, run `tiup list tidb`. This command returns the access methods of the cluster: @@ -168,10 +168,10 @@ As a distributed system, a basic TiDB test cluster usually consists of 2 TiDB in {{< copyable "shell-regular" >}} ```shell - tiup playground v5.4.0 --db 2 --pd 3 --kv 3 + tiup playground v6.0.0 --db 2 --pd 3 --kv 3 ``` - The command downloads a version cluster to the local machine and starts it, such as v5.4.0. To view the latest version, run `tiup list tidb`. + The command downloads a version cluster to the local machine and starts it, such as v6.0.0. To view the latest version, run `tiup list tidb`. This command returns the access methods of the cluster: diff --git a/scale-tidb-using-tiup.md b/scale-tidb-using-tiup.md index 7dab51f609db3..afbcaa1304df9 100644 --- a/scale-tidb-using-tiup.md +++ b/scale-tidb-using-tiup.md @@ -265,9 +265,9 @@ If you want to remove a TiKV node from the `10.0.1.5` host, take the following s ``` ``` - Starting /root/.tiup/components/cluster/v1.9.0/cluster display + Starting /root/.tiup/components/cluster/v1.9.3/cluster display TiDB Cluster: - TiDB Version: v5.4.0 + TiDB Version: v6.0.0 ID Role Host Ports Status Data Dir Deploy Dir -- ---- ---- ----- ------ -------- ---------- 10.0.1.3:8300 cdc 10.0.1.3 8300 Up data/cdc-8300 deploy/cdc-8300 diff --git a/tiup/tiup-bench.md b/tiup/tiup-bench.md index 296b0a078551b..0805cb2f90293 100644 --- a/tiup/tiup-bench.md +++ b/tiup/tiup-bench.md @@ -15,7 +15,6 @@ tiup bench ``` ``` -Starting component `bench`: /home/tidb/.tiup/components/bench/v1.5.0/bench Benchmark database with different workloads Usage: diff --git a/tiup/tiup-cluster.md b/tiup/tiup-cluster.md index 9a4158346569e..1f0e34928c098 100644 --- a/tiup/tiup-cluster.md +++ b/tiup/tiup-cluster.md @@ -17,7 +17,7 @@ tiup cluster ``` ``` -Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.9.1/cluster +Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.9.3/cluster Deploy a TiDB cluster for production Usage: @@ -113,12 +113,12 @@ tidb_servers: ... ``` -Save the file as `/tmp/topology.yaml`. If you want to use TiDB v5.4.0 and your cluster name is `prod-cluster`, run the following command: +Save the file as `/tmp/topology.yaml`. If you want to use TiDB v6.0.0 and your cluster name is `prod-cluster`, run the following command: {{< copyable "shell-regular" >}} ```shell -tiup cluster deploy -p prod-cluster v5.4.0 /tmp/topology.yaml +tiup cluster deploy -p prod-cluster v6.0.0 /tmp/topology.yaml ``` During the execution, TiUP asks you to confirm your topology again and requires the root password of the target machine (the `-p` flag means inputting password): @@ -126,7 +126,7 @@ During the execution, TiUP asks you to confirm your topology again and requires ```bash Please confirm your topology: TiDB Cluster: prod-cluster -TiDB Version: v5.4.0 +TiDB Version: v6.0.0 Type Host Ports Directories ---- ---- ----- ----------- pd 172.16.5.134 2379/2380 deploy/pd-2379,data/pd-2379 @@ -163,10 +163,10 @@ tiup cluster list ``` ``` -Starting /root/.tiup/components/cluster/v1.9.1/cluster list +Starting /root/.tiup/components/cluster/v1.9.3/cluster list Name User Version Path PrivateKey ---- ---- ------- ---- ---------- -prod-cluster tidb v5.4.0 /root/.tiup/storage/cluster/clusters/prod-cluster /root/.tiup/storage/cluster/clusters/prod-cluster/ssh/id_rsa +prod-cluster tidb v6.0.0 /root/.tiup/storage/cluster/clusters/prod-cluster /root/.tiup/storage/cluster/clusters/prod-cluster/ssh/id_rsa ``` ## Start the cluster @@ -192,9 +192,9 @@ tiup cluster display prod-cluster ``` ``` -Starting /root/.tiup/components/cluster/v1.9.1/cluster display prod-cluster +Starting /root/.tiup/components/cluster/v1.9.3/cluster display prod-cluster TiDB Cluster: prod-cluster -TiDB Version: v5.4.0 +TiDB Version: v6.0.0 ID Role Host Ports Status Data Dir Deploy Dir -- ---- ---- ----- ------ -------- ---------- 172.16.5.134:3000 grafana 172.16.5.134 3000 Up - deploy/grafana-3000 @@ -263,9 +263,9 @@ tiup cluster display prod-cluster ``` ``` -Starting /root/.tiup/components/cluster/v1.9.1/cluster display prod-cluster +Starting /root/.tiup/components/cluster/v1.9.3/cluster display prod-cluster TiDB Cluster: prod-cluster -TiDB Version: v5.4.0 +TiDB Version: v6.0.0 ID Role Host Ports Status Data Dir Deploy Dir -- ---- ---- ----- ------ -------- ---------- 172.16.5.134:3000 grafana 172.16.5.134 3000 Up - deploy/grafana-3000 @@ -373,12 +373,12 @@ Global Flags: -y, --yes Skip all confirmations and assumes 'yes' ``` -For example, the following command upgrades the cluster to v5.4.0: +For example, the following command upgrades the cluster to v6.0.0: {{< copyable "shell-regular" >}} ```bash -tiup cluster upgrade tidb-test v5.4.0 +tiup cluster upgrade tidb-test v6.0.0 ``` ## Update configuration @@ -557,14 +557,14 @@ tiup cluster audit ``` ``` -Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.9.1/cluster audit +Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.9.3/cluster audit ID Time Command -- ---- ------- -4BLhr0 2022-03-01T13:25:09+08:00 /home/tidb/.tiup/components/cluster/v1.9.1/cluster deploy test v5.4.0 /tmp/topology.yaml -4BKWjF 2022-02-28T23:36:57+08:00 /home/tidb/.tiup/components/cluster/v1.9.1/cluster deploy test v5.4.0 /tmp/topology.yaml -4BKVwH 2022-02-28T23:02:08+08:00 /home/tidb/.tiup/components/cluster/v1.9.1/cluster deploy test v5.4.0 /tmp/topology.yaml -4BKKH1 2022-02-28T16:39:04+08:00 /home/tidb/.tiup/components/cluster/v1.9.1/cluster destroy test -4BKKDx 2022-02-28T16:36:57+08:00 /home/tidb/.tiup/components/cluster/v1.9.1/cluster deploy test v5.4.0 /tmp/topology.yaml +4BLhr0 2022-03-01T13:25:09+08:00 /home/tidb/.tiup/components/cluster/v1.9.3/cluster deploy test v6.0.0 /tmp/topology.yaml +4BKWjF 2022-02-28T23:36:57+08:00 /home/tidb/.tiup/components/cluster/v1.9.3/cluster deploy test v6.0.0 /tmp/topology.yaml +4BKVwH 2022-02-28T23:02:08+08:00 /home/tidb/.tiup/components/cluster/v1.9.3/cluster deploy test v6.0.0 /tmp/topology.yaml +4BKKH1 2022-02-28T16:39:04+08:00 /home/tidb/.tiup/components/cluster/v1.9.3/cluster destroy test +4BKKDx 2022-02-28T16:36:57+08:00 /home/tidb/.tiup/components/cluster/v1.9.3/cluster deploy test v6.0.0 /tmp/topology.yaml ``` The first column is `audit-id`. To view the execution log of a certain command, pass the `audit-id` of a command as the flag as follows: diff --git a/tiup/tiup-component-cluster-deploy.md b/tiup/tiup-component-cluster-deploy.md index 9438ae2aaa596..c9c2852c0ea71 100644 --- a/tiup/tiup-component-cluster-deploy.md +++ b/tiup/tiup-component-cluster-deploy.md @@ -13,7 +13,7 @@ tiup cluster deploy [flags] ``` - ``: the name of the new cluster, which cannot be the same as the existing cluster names. -- ``: the version number of the TiDB cluster to deploy, such as `v5.4.0`. +- ``: the version number of the TiDB cluster to deploy, such as `v6.0.0`. - ``: the prepared [topology file](/tiup/tiup-cluster-topology-reference.md). ## Options diff --git a/tiup/tiup-component-management.md b/tiup/tiup-component-management.md index 0dda75994676c..8e1747150139b 100644 --- a/tiup/tiup-component-management.md +++ b/tiup/tiup-component-management.md @@ -70,12 +70,12 @@ Example 2: Use TiUP to install the nightly version of TiDB. tiup install tidb:nightly ``` -Example 3: Use TiUP to install TiKV v5.4.0. +Example 3: Use TiUP to install TiKV v6.0.0. {{< copyable "shell-regular" >}} ```shell -tiup install tikv:v5.4.0 +tiup install tikv:v6.0.0 ``` ## Upgrade components @@ -128,12 +128,12 @@ Before the component is started, TiUP creates a directory for it, and then puts If you want to start the same component multiple times and reuse the previous working directory, you can use `--tag` to specify the same name when the component is started. After the tag is specified, the working directory will *not be automatically deleted* when the instance is terminated, which makes it convenient to reuse the working directory. -Example 1: Operate TiDB v5.4.0. +Example 1: Operate TiDB v6.0.0. {{< copyable "shell-regular" >}} ```shell -tiup tidb:v5.4.0 +tiup tidb:v6.0.0 ``` Example 2: Specify the tag with which TiKV operates. @@ -219,12 +219,12 @@ The following flags are supported in this command: - If the version is ignored, adding `--all` means to uninstall all versions of this component. - If the version and the component are both ignored, adding `--all` means to uninstall all components of all versions. -Example 1: Uninstall TiDB v5.4.0. +Example 1: Uninstall TiDB v6.0.0. {{< copyable "shell-regular" >}} ```shell -tiup uninstall tidb:v5.4.0 +tiup uninstall tidb:v6.0.0 ``` Example 2: Uninstall TiKV of all versions. diff --git a/tiup/tiup-mirror.md b/tiup/tiup-mirror.md index 2b830d94a9a84..6e898ba8872ce 100644 --- a/tiup/tiup-mirror.md +++ b/tiup/tiup-mirror.md @@ -88,9 +88,9 @@ The `tiup mirror clone` command provides many optional flags (might provide more If you want to clone only one version (not all versions) of a component, use `--=` to specify this version. For example: - - Execute the `tiup mirror clone --tidb v5.4.0` command to clone the v5.4.0 version of the TiDB component. - - Run the `tiup mirror clone --tidb v5.4.0 --tikv all` command to clone the v5.4.0 version of the TiDB component and all versions of the TiKV component. - - Run the `tiup mirror clone v5.4.0` command to clone the v5.4.0 version of all components in a cluster. + - Execute the `tiup mirror clone --tidb v6.0.0` command to clone the v6.0.0 version of the TiDB component. + - Run the `tiup mirror clone --tidb v6.0.0 --tikv all` command to clone the v6.0.0 version of the TiDB component and all versions of the TiKV component. + - Run the `tiup mirror clone v6.0.0` command to clone the v6.0.0 version of all components in a cluster. After cloning, signing keys are set up automatically. diff --git a/tiup/tiup-playground.md b/tiup/tiup-playground.md index 6d194428e2f9f..3b72b4206b6c6 100644 --- a/tiup/tiup-playground.md +++ b/tiup/tiup-playground.md @@ -20,9 +20,9 @@ If you directly execute the `tiup playground` command, TiUP uses the locally ins This command actually performs the following operations: -- Because this command does not specify the version of the playground component, TiUP first checks the latest version of the installed playground component. Assume that the latest version is v1.9.0, then this command works the same as `tiup playground:v1.9.0`. +- Because this command does not specify the version of the playground component, TiUP first checks the latest version of the installed playground component. Assume that the latest version is v1.9.3, then this command works the same as `tiup playground:v1.9.3`. - If you have not used TiUP playground to install the TiDB, TiKV, and PD components, the playground component installs the latest stable version of these components, and then start these instances. -- Because this command does not specify the version of the TiDB, PD, and TiKV component, TiUP playground uses the latest version of each component by default. Assume that the latest version is v5.4.0, then this command works the same as `tiup playground:v1.9.0 v5.4.0`. +- Because this command does not specify the version of the TiDB, PD, and TiKV component, TiUP playground uses the latest version of each component by default. Assume that the latest version is v6.0.0, then this command works the same as `tiup playground:v1.9.3 v6.0.0`. - Because this command does not specify the number of each component, TiUP playground, by default, starts a smallest cluster that consists of one TiDB instance, one TiKV instance, one PD instance, and one TiFlash instance. - After starting each TiDB component, TiUP playground reminds you that the cluster is successfully started and provides you some useful information, such as how to connect to the TiDB cluster through the MySQL client and how to access the [TiDB Dashboard](/dashboard/dashboard-intro.md). diff --git a/upgrade-tidb-using-tiup.md b/upgrade-tidb-using-tiup.md index 8bd0c4cfe401e..653db1cd803e5 100644 --- a/upgrade-tidb-using-tiup.md +++ b/upgrade-tidb-using-tiup.md @@ -8,25 +8,22 @@ aliases: ['/docs/dev/upgrade-tidb-using-tiup/','/docs/dev/how-to/upgrade/using-t This document is targeted for the following upgrade paths: -- Upgrade from TiDB 4.0 versions to TiDB 5.4 versions. -- Upgrade from TiDB 5.0 versions to TiDB 5.4 versions. -- Upgrade from TiDB 5.1 versions to TiDB 5.4 versions. -- Upgrade from TiDB 5.2 versions to TiDB 5.4 versions. -- Upgrade from TiDB 5.3 versions to TiDB 5.4 versions. +- Upgrade from TiDB 4.0 versions to TiDB 6.0. +- Upgrade from TiDB 5.0-5.4 versions to TiDB 6.0. > **Note:** > -> If your cluster to be upgraded is v3.1 or an earlier version (v3.0 or v2.1), the direct upgrade to v5.4 or its patch versions is not supported. You need to upgrade your cluster first to v4.0 and then to v5.4. +> If your cluster to be upgraded is v3.1 or an earlier version (v3.0 or v2.1), the direct upgrade to v6.0 or its patch versions is not supported. You need to upgrade your cluster first to v4.0 and then to v6.0. ## Upgrade caveat - TiDB currently does not support version downgrade or rolling back to an earlier version after the upgrade. -- For the v4.0 cluster managed using TiDB Ansible, you need to import the cluster to TiUP (`tiup cluster`) for new management according to [Upgrade TiDB Using TiUP (v4.0)](https://docs.pingcap.com/tidb/v4.0/upgrade-tidb-using-tiup#import-tidb-ansible-and-the-inventoryini-configuration-to-tiup). Then you can upgrade the cluster to v5.4 or its patch versions according to this document. -- To update versions earlier than 3.0 to 5.4: +- For the v4.0 cluster managed using TiDB Ansible, you need to import the cluster to TiUP (`tiup cluster`) for new management according to [Upgrade TiDB Using TiUP (v4.0)](https://docs.pingcap.com/tidb/v4.0/upgrade-tidb-using-tiup#import-tidb-ansible-and-the-inventoryini-configuration-to-tiup). Then you can upgrade the cluster to v6.0 or its patch versions according to this document. +- To update versions earlier than 3.0 to 6.0: 1. Update this version to 3.0 using [TiDB Ansible](https://docs.pingcap.com/tidb/v3.0/upgrade-tidb-using-ansible). 2. Use TiUP (`tiup cluster`) to import the TiDB Ansible configuration. 3. Update the 3.0 version to 4.0 according to [Upgrade TiDB Using TiUP (v4.0)](https://docs.pingcap.com/tidb/v4.0/upgrade-tidb-using-tiup#import-tidb-ansible-and-the-inventoryini-configuration-to-tiup). - 4. Upgrade the cluster to v5.4 according to this document. + 4. Upgrade the cluster to v6.0 according to this document. - Support upgrading the versions of TiDB Binlog, TiCDC, TiFlash, and other components. - For detailed compatibility changes of different versions, see the [Release Notes](/releases/release-notes.md) of each version. Modify your cluster configuration according to the "Compatibility Changes" section of the corresponding release notes. @@ -50,7 +47,7 @@ Before upgrading your TiDB cluster, you first need to upgrade TiUP or TiUP mirro > > If the control machine of the cluster to upgrade cannot access `https://tiup-mirrors.pingcap.com`, skip this section and see [Upgrade TiUP offline mirror](#upgrade-tiup-offline-mirror). -1. Upgrade the TiUP version. It is recommended that the TiUP version is `1.9.0` or later. +1. Upgrade the TiUP version. It is recommended that the TiUP version is `1.9.3` or later. {{< copyable "shell-regular" >}} @@ -59,7 +56,7 @@ Before upgrading your TiDB cluster, you first need to upgrade TiUP or TiUP mirro tiup --version ``` -2. Upgrade the TiUP Cluster version. It is recommended that the TiUP Cluster version is `1.9.0` or later. +2. Upgrade the TiUP Cluster version. It is recommended that the TiUP Cluster version is `1.9.3` or later. {{< copyable "shell-regular" >}} @@ -101,7 +98,7 @@ Now, the offline mirror has been upgraded successfully. If an error occurs durin > Skip this step if one of the following situations applies: > > + You have not modified the configuration parameters of the original cluster. Or you have modified the configuration parameters using `tiup cluster` but no more modification is needed. -> + After the upgrade, you want to use v5.4's default parameter values for the unmodified configuration items. +> + After the upgrade, you want to use v6.0's default parameter values for the unmodified configuration items. 1. Enter the `vi` editing mode to edit the topology file: @@ -117,7 +114,7 @@ Now, the offline mirror has been upgraded successfully. If an error occurs durin > **Note:** > -> Before you upgrade the cluster to v5.4, make sure that the parameters you have modified in v4.0 are compatible in v5.4. For details, see [TiKV Configuration File](/tikv-configuration-file.md). +> Before you upgrade the cluster to v6.0, make sure that the parameters you have modified in v4.0 are compatible in v6.0. For details, see [TiKV Configuration File](/tikv-configuration-file.md). > > The following three TiKV parameters are obsolete in TiDB v5.0. If the following parameters have been configured in your original cluster, you need to delete these parameters through `edit-config`: > @@ -160,12 +157,12 @@ If your application has a maintenance window for the database to be stopped for tiup cluster upgrade ``` -For example, if you want to upgrade the cluster to v5.4.0: +For example, if you want to upgrade the cluster to v6.0.0: {{< copyable "shell-regular" >}} ```shell -tiup cluster upgrade v5.4.0 +tiup cluster upgrade v6.0.0 ``` > **Note:** @@ -215,7 +212,7 @@ tiup cluster display ``` Cluster type: tidb Cluster name: -Cluster version: v5.4.0 +Cluster version: v6.0.0 ``` > **Note:** @@ -265,10 +262,10 @@ You can upgrade the tool version by using TiUP to install the `ctl` component of {{< copyable "shell-regular" >}} ```shell -tiup install ctl:v5.4.0 +tiup install ctl:v6.0.0 ``` -## TiDB 5.4 compatibility changes +## TiDB 6.0 compatibility changes -- See TiDB 5.4 Release Notes for the compatibility changes. +- See TiDB 6.0 Release Notes for the compatibility changes. - Try to avoid creating a new clustered index table when you apply rolling updates to the clusters using TiDB Binlog. From 6ed54a9ff4c7201d9735745099ab28d016c10492 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Sat, 2 Apr 2022 11:58:29 +0800 Subject: [PATCH 32/36] bump version for v6.0-ecosystem tools and PD (#8039) --- download-ecosystem-tools.md | 10 +++++----- pd-control.md | 2 +- pd-recover.md | 2 +- ticdc/manage-ticdc.md | 4 ++-- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/download-ecosystem-tools.md b/download-ecosystem-tools.md index e0c50497eec8c..7988755c568a0 100644 --- a/download-ecosystem-tools.md +++ b/download-ecosystem-tools.md @@ -26,7 +26,7 @@ If you want to download the latest version of [TiDB Binlog](/tidb-binlog/tidb-bi > **Note:** > -> `{version}` in the above download link indicates the version number of TiDB. For example, the download link for `v5.4.0` is `https://download.pingcap.org/tidb-v5.4.0-linux-amd64.tar.gz`. +> `{version}` in the above download link indicates the version number of TiDB. For example, the download link for `v6.0.0` is `https://download.pingcap.org/tidb-v6.0.0-linux-amd64.tar.gz`. ## TiDB Lightning @@ -38,7 +38,7 @@ Download [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) by using t > **Note:** > -> `{version}` in the above download link indicates the version number of TiDB Lightning. For example, the download link for `v5.4.0` is `https://download.pingcap.org/tidb-toolkit-v5.4.0-linux-amd64.tar.gz`. +> `{version}` in the above download link indicates the version number of TiDB Lightning. For example, the download link for `v6.0.0` is `https://download.pingcap.org/tidb-toolkit-v6.0.0-linux-amd64.tar.gz`. ## BR (backup and restore) @@ -50,7 +50,7 @@ Download [BR](/br/backup-and-restore-tool.md) by using the download link in the > **Note:** > -> `{version}` in the above download link indicates the version number of BR. For example, the download link for `v5.4.0` is `https://download.pingcap.org/tidb-toolkit-v5.4.0-linux-amd64.tar.gz`. +> `{version}` in the above download link indicates the version number of BR. For example, the download link for `v6.0.0` is `https://download.pingcap.org/tidb-toolkit-v6.0.0-linux-amd64.tar.gz`. ## TiDB DM (Data Migration) @@ -62,7 +62,7 @@ Download [DM](/dm/dm-overview.md) by using the download link in the following ta > **Note:** > -> `{version}` in the above download link indicates the version number of DM. For example, the download link for `v5.4.0` is `https://download.pingcap.org/dm-v5.4.0-linux-amd64.tar.gz`. You can check the published DM versions in the [DM Release](https://github.com/pingcap/dm/releases) page. +> `{version}` in the above download link indicates the version number of DM. For example, the download link for `v6.0.0` is `https://download.pingcap.org/dm-v6.0.0-linux-amd64.tar.gz`. You can check the published DM versions in the [DM Release](https://github.com/pingcap/dm/releases) page. ## Dumpling @@ -74,7 +74,7 @@ Download [Dumpling](/dumpling-overview.md) from the links below: > **Note:** > -> The `{version}` in the download link is the version number of Dumpling. For example, the link for downloading the `v5.4.0` version of Dumpling is `https://download.pingcap.org/tidb-toolkit-v5.4.0-linux-amd64.tar.gz`. You can view the currently released versions in [TiDB Releases](https://github.com/pingcap/tidb/releases). +> The `{version}` in the download link is the version number of Dumpling. For example, the link for downloading the `v6.0.0` version of Dumpling is `https://download.pingcap.org/tidb-toolkit-v6.0.0-linux-amd64.tar.gz`. You can view the currently released versions in [TiDB Releases](https://github.com/pingcap/tidb/releases). > > Dumpling supports arm64 linux. You can replace `amd64` in the download link with `arm64`, which means the `arm64` version of Dumpling. diff --git a/pd-control.md b/pd-control.md index 10281b14dfbdd..e00ed094bdadb 100644 --- a/pd-control.md +++ b/pd-control.md @@ -28,7 +28,7 @@ If you want to download the latest version of `pd-ctl`, directly download the Ti > **Note:** > -> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.4.0`, the package download link is `https://download.pingcap.org/tidb-v5.4.0-linux-amd64.tar.gz`. +> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v6.0.0`, the package download link is `https://download.pingcap.org/tidb-v6.0.0-linux-amd64.tar.gz`. ### Compile from source code diff --git a/pd-recover.md b/pd-recover.md index 1995cc7339648..0f7ded81da074 100644 --- a/pd-recover.md +++ b/pd-recover.md @@ -27,7 +27,7 @@ To download the latest version of PD Recover, directly download the TiDB package > **Note:** > -> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.4.0`, the package download link is `https://download.pingcap.org/tidb-v5.4.0-linux-amd64.tar.gz`. +> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v6.0.0`, the package download link is `https://download.pingcap.org/tidb-v6.0.0-linux-amd64.tar.gz`. ## Quick Start diff --git a/ticdc/manage-ticdc.md b/ticdc/manage-ticdc.md index 54144f585db7b..8311fcfaf0945 100644 --- a/ticdc/manage-ticdc.md +++ b/ticdc/manage-ticdc.md @@ -12,14 +12,14 @@ You can also use the HTTP interface (the TiCDC OpenAPI feature) to manage the Ti ## Upgrade TiCDC using TiUP -This section introduces how to upgrade the TiCDC cluster using TiUP. In the following example, assume that you need to upgrade TiCDC and the entire TiDB cluster to v5.4.0. +This section introduces how to upgrade the TiCDC cluster using TiUP. In the following example, assume that you need to upgrade TiCDC and the entire TiDB cluster to v6.0.0. {{< copyable "shell-regular" >}} ```shell tiup update --self && \ tiup update --all && \ -tiup cluster upgrade v5.4.0 +tiup cluster upgrade v6.0.0 ``` ### Notes for upgrade From 5b0234d4dd787039b953d911e09ceef980d0e308 Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Sat, 2 Apr 2022 12:05:27 +0800 Subject: [PATCH 33/36] Revert "Merge branch 'master' into pr/8037" This reverts commit 168b598b6b0694563d66407362e471dd9a9d995c, reversing changes made to 4bef2ffb806c2328554c3c7ac0f09106df5410f1. --- download-ecosystem-tools.md | 10 ++++---- pd-control.md | 2 +- pd-recover.md | 2 +- production-deployment-using-tiup.md | 12 ++++----- quick-start-with-tidb.md | 8 +++--- scale-tidb-using-tiup.md | 4 +-- ticdc/manage-ticdc.md | 4 +-- tiup/tiup-bench.md | 1 + tiup/tiup-cluster.md | 36 +++++++++++++-------------- tiup/tiup-component-cluster-deploy.md | 2 +- tiup/tiup-component-management.md | 12 ++++----- tiup/tiup-mirror.md | 6 ++--- tiup/tiup-playground.md | 4 +-- upgrade-tidb-using-tiup.md | 35 ++++++++++++++------------ 14 files changed, 71 insertions(+), 67 deletions(-) diff --git a/download-ecosystem-tools.md b/download-ecosystem-tools.md index 7988755c568a0..e0c50497eec8c 100644 --- a/download-ecosystem-tools.md +++ b/download-ecosystem-tools.md @@ -26,7 +26,7 @@ If you want to download the latest version of [TiDB Binlog](/tidb-binlog/tidb-bi > **Note:** > -> `{version}` in the above download link indicates the version number of TiDB. For example, the download link for `v6.0.0` is `https://download.pingcap.org/tidb-v6.0.0-linux-amd64.tar.gz`. +> `{version}` in the above download link indicates the version number of TiDB. For example, the download link for `v5.4.0` is `https://download.pingcap.org/tidb-v5.4.0-linux-amd64.tar.gz`. ## TiDB Lightning @@ -38,7 +38,7 @@ Download [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) by using t > **Note:** > -> `{version}` in the above download link indicates the version number of TiDB Lightning. For example, the download link for `v6.0.0` is `https://download.pingcap.org/tidb-toolkit-v6.0.0-linux-amd64.tar.gz`. +> `{version}` in the above download link indicates the version number of TiDB Lightning. For example, the download link for `v5.4.0` is `https://download.pingcap.org/tidb-toolkit-v5.4.0-linux-amd64.tar.gz`. ## BR (backup and restore) @@ -50,7 +50,7 @@ Download [BR](/br/backup-and-restore-tool.md) by using the download link in the > **Note:** > -> `{version}` in the above download link indicates the version number of BR. For example, the download link for `v6.0.0` is `https://download.pingcap.org/tidb-toolkit-v6.0.0-linux-amd64.tar.gz`. +> `{version}` in the above download link indicates the version number of BR. For example, the download link for `v5.4.0` is `https://download.pingcap.org/tidb-toolkit-v5.4.0-linux-amd64.tar.gz`. ## TiDB DM (Data Migration) @@ -62,7 +62,7 @@ Download [DM](/dm/dm-overview.md) by using the download link in the following ta > **Note:** > -> `{version}` in the above download link indicates the version number of DM. For example, the download link for `v6.0.0` is `https://download.pingcap.org/dm-v6.0.0-linux-amd64.tar.gz`. You can check the published DM versions in the [DM Release](https://github.com/pingcap/dm/releases) page. +> `{version}` in the above download link indicates the version number of DM. For example, the download link for `v5.4.0` is `https://download.pingcap.org/dm-v5.4.0-linux-amd64.tar.gz`. You can check the published DM versions in the [DM Release](https://github.com/pingcap/dm/releases) page. ## Dumpling @@ -74,7 +74,7 @@ Download [Dumpling](/dumpling-overview.md) from the links below: > **Note:** > -> The `{version}` in the download link is the version number of Dumpling. For example, the link for downloading the `v6.0.0` version of Dumpling is `https://download.pingcap.org/tidb-toolkit-v6.0.0-linux-amd64.tar.gz`. You can view the currently released versions in [TiDB Releases](https://github.com/pingcap/tidb/releases). +> The `{version}` in the download link is the version number of Dumpling. For example, the link for downloading the `v5.4.0` version of Dumpling is `https://download.pingcap.org/tidb-toolkit-v5.4.0-linux-amd64.tar.gz`. You can view the currently released versions in [TiDB Releases](https://github.com/pingcap/tidb/releases). > > Dumpling supports arm64 linux. You can replace `amd64` in the download link with `arm64`, which means the `arm64` version of Dumpling. diff --git a/pd-control.md b/pd-control.md index e00ed094bdadb..10281b14dfbdd 100644 --- a/pd-control.md +++ b/pd-control.md @@ -28,7 +28,7 @@ If you want to download the latest version of `pd-ctl`, directly download the Ti > **Note:** > -> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v6.0.0`, the package download link is `https://download.pingcap.org/tidb-v6.0.0-linux-amd64.tar.gz`. +> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.4.0`, the package download link is `https://download.pingcap.org/tidb-v5.4.0-linux-amd64.tar.gz`. ### Compile from source code diff --git a/pd-recover.md b/pd-recover.md index 0f7ded81da074..1995cc7339648 100644 --- a/pd-recover.md +++ b/pd-recover.md @@ -27,7 +27,7 @@ To download the latest version of PD Recover, directly download the TiDB package > **Note:** > -> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v6.0.0`, the package download link is `https://download.pingcap.org/tidb-v6.0.0-linux-amd64.tar.gz`. +> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.4.0`, the package download link is `https://download.pingcap.org/tidb-v5.4.0-linux-amd64.tar.gz`. ## Quick Start diff --git a/production-deployment-using-tiup.md b/production-deployment-using-tiup.md index 90c1adfc200d4..9836b27743729 100644 --- a/production-deployment-using-tiup.md +++ b/production-deployment-using-tiup.md @@ -141,12 +141,12 @@ To prepare the TiUP offline component package, manually pack an offline componen If you want to adjust an existing offline mirror (such as adding a new version of a component), take the following steps: - 1. When pulling an offline mirror, you can get an incomplete offline mirror by specifying specific information via parameters, such as the component and version information. For example, you can pull an offline mirror that includes only the offline mirror of TiUP v1.9.3 and TiUP Cluster v1.9.3 by running the following command: + 1. When pulling an offline mirror, you can get an incomplete offline mirror by specifying specific information via parameters, such as the component and version information. For example, you can pull an offline mirror that includes only the offline mirror of TiUP v1.9.0 and TiUP Cluster v1.9.0 by running the following command: {{< copyable "shell-regular" >}} ```bash - tiup mirror clone tiup-custom-mirror-v1.9.3 --tiup v1.9.3 --cluster v1.9.3 + tiup mirror clone tiup-custom-mirror-v1.9.0 --tiup v1.9.0 --cluster v1.9.0 ``` If you only need the components for a particular platform, you can specify them using the `--os` or `--arch` parameters. @@ -178,10 +178,10 @@ To prepare the TiUP offline component package, manually pack an offline componen {{< copyable "shell-regular" >}} ```bash - tiup mirror merge tiup-custom-mirror-v1.9.3 + tiup mirror merge tiup-custom-mirror-v1.9.0 ``` - 5. When the above steps are completed, check the result by running the `tiup list` command. In this document's example, the outputs of both `tiup list tiup` and `tiup list cluster` show that the corresponding components of `v1.9.3` are available. + 5. When the above steps are completed, check the result by running the `tiup list` command. In this document's example, the outputs of both `tiup list tiup` and `tiup list cluster` show that the corresponding components of `v1.9.0` are available. #### Step 2: Deploy the offline TiUP component @@ -310,13 +310,13 @@ Then execute the `deploy` command to deploy the TiDB cluster: {{< copyable "shell-regular" >}} ```shell -tiup cluster deploy tidb-test v6.0.0 ./topology.yaml --user root [-p] [-i /home/root/.ssh/gcp_rsa] +tiup cluster deploy tidb-test v5.4.0 ./topology.yaml --user root [-p] [-i /home/root/.ssh/gcp_rsa] ``` In the above command: - The name of the deployed TiDB cluster is `tidb-test`. -- You can see the latest supported versions by running `tiup list tidb`. This document takes `v6.0.0` as an example. +- You can see the latest supported versions by running `tiup list tidb`. This document takes `v5.4.0` as an example. - The initialization configuration file is `topology.yaml`. - `--user root`: Log in to the target machine through the `root` key to complete the cluster deployment, or you can use other users with `ssh` and `sudo` privileges to complete the deployment. - `[-i]` and `[-p]`: optional. If you have configured login to the target machine without password, these parameters are not required. If not, choose one of the two parameters. `[-i]` is the private key of the `root` user (or other users specified by `--user`) that has access to the target machine. `[-p]` is used to input the user password interactively. diff --git a/quick-start-with-tidb.md b/quick-start-with-tidb.md index d5b27dd689662..7d090bc96fc03 100644 --- a/quick-start-with-tidb.md +++ b/quick-start-with-tidb.md @@ -65,10 +65,10 @@ As a distributed system, a basic TiDB test cluster usually consists of 2 TiDB in {{< copyable "shell-regular" >}} ```shell - tiup playground v6.0.0 --db 2 --pd 3 --kv 3 + tiup playground v5.4.0 --db 2 --pd 3 --kv 3 ``` - The command downloads a version cluster to the local machine and starts it, such as v6.0.0. To view the latest version, run `tiup list tidb`. + The command downloads a version cluster to the local machine and starts it, such as v5.4.0. To view the latest version, run `tiup list tidb`. This command returns the access methods of the cluster: @@ -168,10 +168,10 @@ As a distributed system, a basic TiDB test cluster usually consists of 2 TiDB in {{< copyable "shell-regular" >}} ```shell - tiup playground v6.0.0 --db 2 --pd 3 --kv 3 + tiup playground v5.4.0 --db 2 --pd 3 --kv 3 ``` - The command downloads a version cluster to the local machine and starts it, such as v6.0.0. To view the latest version, run `tiup list tidb`. + The command downloads a version cluster to the local machine and starts it, such as v5.4.0. To view the latest version, run `tiup list tidb`. This command returns the access methods of the cluster: diff --git a/scale-tidb-using-tiup.md b/scale-tidb-using-tiup.md index afbcaa1304df9..7dab51f609db3 100644 --- a/scale-tidb-using-tiup.md +++ b/scale-tidb-using-tiup.md @@ -265,9 +265,9 @@ If you want to remove a TiKV node from the `10.0.1.5` host, take the following s ``` ``` - Starting /root/.tiup/components/cluster/v1.9.3/cluster display + Starting /root/.tiup/components/cluster/v1.9.0/cluster display TiDB Cluster: - TiDB Version: v6.0.0 + TiDB Version: v5.4.0 ID Role Host Ports Status Data Dir Deploy Dir -- ---- ---- ----- ------ -------- ---------- 10.0.1.3:8300 cdc 10.0.1.3 8300 Up data/cdc-8300 deploy/cdc-8300 diff --git a/ticdc/manage-ticdc.md b/ticdc/manage-ticdc.md index 8311fcfaf0945..54144f585db7b 100644 --- a/ticdc/manage-ticdc.md +++ b/ticdc/manage-ticdc.md @@ -12,14 +12,14 @@ You can also use the HTTP interface (the TiCDC OpenAPI feature) to manage the Ti ## Upgrade TiCDC using TiUP -This section introduces how to upgrade the TiCDC cluster using TiUP. In the following example, assume that you need to upgrade TiCDC and the entire TiDB cluster to v6.0.0. +This section introduces how to upgrade the TiCDC cluster using TiUP. In the following example, assume that you need to upgrade TiCDC and the entire TiDB cluster to v5.4.0. {{< copyable "shell-regular" >}} ```shell tiup update --self && \ tiup update --all && \ -tiup cluster upgrade v6.0.0 +tiup cluster upgrade v5.4.0 ``` ### Notes for upgrade diff --git a/tiup/tiup-bench.md b/tiup/tiup-bench.md index 0805cb2f90293..296b0a078551b 100644 --- a/tiup/tiup-bench.md +++ b/tiup/tiup-bench.md @@ -15,6 +15,7 @@ tiup bench ``` ``` +Starting component `bench`: /home/tidb/.tiup/components/bench/v1.5.0/bench Benchmark database with different workloads Usage: diff --git a/tiup/tiup-cluster.md b/tiup/tiup-cluster.md index 1f0e34928c098..9a4158346569e 100644 --- a/tiup/tiup-cluster.md +++ b/tiup/tiup-cluster.md @@ -17,7 +17,7 @@ tiup cluster ``` ``` -Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.9.3/cluster +Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.9.1/cluster Deploy a TiDB cluster for production Usage: @@ -113,12 +113,12 @@ tidb_servers: ... ``` -Save the file as `/tmp/topology.yaml`. If you want to use TiDB v6.0.0 and your cluster name is `prod-cluster`, run the following command: +Save the file as `/tmp/topology.yaml`. If you want to use TiDB v5.4.0 and your cluster name is `prod-cluster`, run the following command: {{< copyable "shell-regular" >}} ```shell -tiup cluster deploy -p prod-cluster v6.0.0 /tmp/topology.yaml +tiup cluster deploy -p prod-cluster v5.4.0 /tmp/topology.yaml ``` During the execution, TiUP asks you to confirm your topology again and requires the root password of the target machine (the `-p` flag means inputting password): @@ -126,7 +126,7 @@ During the execution, TiUP asks you to confirm your topology again and requires ```bash Please confirm your topology: TiDB Cluster: prod-cluster -TiDB Version: v6.0.0 +TiDB Version: v5.4.0 Type Host Ports Directories ---- ---- ----- ----------- pd 172.16.5.134 2379/2380 deploy/pd-2379,data/pd-2379 @@ -163,10 +163,10 @@ tiup cluster list ``` ``` -Starting /root/.tiup/components/cluster/v1.9.3/cluster list +Starting /root/.tiup/components/cluster/v1.9.1/cluster list Name User Version Path PrivateKey ---- ---- ------- ---- ---------- -prod-cluster tidb v6.0.0 /root/.tiup/storage/cluster/clusters/prod-cluster /root/.tiup/storage/cluster/clusters/prod-cluster/ssh/id_rsa +prod-cluster tidb v5.4.0 /root/.tiup/storage/cluster/clusters/prod-cluster /root/.tiup/storage/cluster/clusters/prod-cluster/ssh/id_rsa ``` ## Start the cluster @@ -192,9 +192,9 @@ tiup cluster display prod-cluster ``` ``` -Starting /root/.tiup/components/cluster/v1.9.3/cluster display prod-cluster +Starting /root/.tiup/components/cluster/v1.9.1/cluster display prod-cluster TiDB Cluster: prod-cluster -TiDB Version: v6.0.0 +TiDB Version: v5.4.0 ID Role Host Ports Status Data Dir Deploy Dir -- ---- ---- ----- ------ -------- ---------- 172.16.5.134:3000 grafana 172.16.5.134 3000 Up - deploy/grafana-3000 @@ -263,9 +263,9 @@ tiup cluster display prod-cluster ``` ``` -Starting /root/.tiup/components/cluster/v1.9.3/cluster display prod-cluster +Starting /root/.tiup/components/cluster/v1.9.1/cluster display prod-cluster TiDB Cluster: prod-cluster -TiDB Version: v6.0.0 +TiDB Version: v5.4.0 ID Role Host Ports Status Data Dir Deploy Dir -- ---- ---- ----- ------ -------- ---------- 172.16.5.134:3000 grafana 172.16.5.134 3000 Up - deploy/grafana-3000 @@ -373,12 +373,12 @@ Global Flags: -y, --yes Skip all confirmations and assumes 'yes' ``` -For example, the following command upgrades the cluster to v6.0.0: +For example, the following command upgrades the cluster to v5.4.0: {{< copyable "shell-regular" >}} ```bash -tiup cluster upgrade tidb-test v6.0.0 +tiup cluster upgrade tidb-test v5.4.0 ``` ## Update configuration @@ -557,14 +557,14 @@ tiup cluster audit ``` ``` -Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.9.3/cluster audit +Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.9.1/cluster audit ID Time Command -- ---- ------- -4BLhr0 2022-03-01T13:25:09+08:00 /home/tidb/.tiup/components/cluster/v1.9.3/cluster deploy test v6.0.0 /tmp/topology.yaml -4BKWjF 2022-02-28T23:36:57+08:00 /home/tidb/.tiup/components/cluster/v1.9.3/cluster deploy test v6.0.0 /tmp/topology.yaml -4BKVwH 2022-02-28T23:02:08+08:00 /home/tidb/.tiup/components/cluster/v1.9.3/cluster deploy test v6.0.0 /tmp/topology.yaml -4BKKH1 2022-02-28T16:39:04+08:00 /home/tidb/.tiup/components/cluster/v1.9.3/cluster destroy test -4BKKDx 2022-02-28T16:36:57+08:00 /home/tidb/.tiup/components/cluster/v1.9.3/cluster deploy test v6.0.0 /tmp/topology.yaml +4BLhr0 2022-03-01T13:25:09+08:00 /home/tidb/.tiup/components/cluster/v1.9.1/cluster deploy test v5.4.0 /tmp/topology.yaml +4BKWjF 2022-02-28T23:36:57+08:00 /home/tidb/.tiup/components/cluster/v1.9.1/cluster deploy test v5.4.0 /tmp/topology.yaml +4BKVwH 2022-02-28T23:02:08+08:00 /home/tidb/.tiup/components/cluster/v1.9.1/cluster deploy test v5.4.0 /tmp/topology.yaml +4BKKH1 2022-02-28T16:39:04+08:00 /home/tidb/.tiup/components/cluster/v1.9.1/cluster destroy test +4BKKDx 2022-02-28T16:36:57+08:00 /home/tidb/.tiup/components/cluster/v1.9.1/cluster deploy test v5.4.0 /tmp/topology.yaml ``` The first column is `audit-id`. To view the execution log of a certain command, pass the `audit-id` of a command as the flag as follows: diff --git a/tiup/tiup-component-cluster-deploy.md b/tiup/tiup-component-cluster-deploy.md index c9c2852c0ea71..9438ae2aaa596 100644 --- a/tiup/tiup-component-cluster-deploy.md +++ b/tiup/tiup-component-cluster-deploy.md @@ -13,7 +13,7 @@ tiup cluster deploy [flags] ``` - ``: the name of the new cluster, which cannot be the same as the existing cluster names. -- ``: the version number of the TiDB cluster to deploy, such as `v6.0.0`. +- ``: the version number of the TiDB cluster to deploy, such as `v5.4.0`. - ``: the prepared [topology file](/tiup/tiup-cluster-topology-reference.md). ## Options diff --git a/tiup/tiup-component-management.md b/tiup/tiup-component-management.md index 8e1747150139b..0dda75994676c 100644 --- a/tiup/tiup-component-management.md +++ b/tiup/tiup-component-management.md @@ -70,12 +70,12 @@ Example 2: Use TiUP to install the nightly version of TiDB. tiup install tidb:nightly ``` -Example 3: Use TiUP to install TiKV v6.0.0. +Example 3: Use TiUP to install TiKV v5.4.0. {{< copyable "shell-regular" >}} ```shell -tiup install tikv:v6.0.0 +tiup install tikv:v5.4.0 ``` ## Upgrade components @@ -128,12 +128,12 @@ Before the component is started, TiUP creates a directory for it, and then puts If you want to start the same component multiple times and reuse the previous working directory, you can use `--tag` to specify the same name when the component is started. After the tag is specified, the working directory will *not be automatically deleted* when the instance is terminated, which makes it convenient to reuse the working directory. -Example 1: Operate TiDB v6.0.0. +Example 1: Operate TiDB v5.4.0. {{< copyable "shell-regular" >}} ```shell -tiup tidb:v6.0.0 +tiup tidb:v5.4.0 ``` Example 2: Specify the tag with which TiKV operates. @@ -219,12 +219,12 @@ The following flags are supported in this command: - If the version is ignored, adding `--all` means to uninstall all versions of this component. - If the version and the component are both ignored, adding `--all` means to uninstall all components of all versions. -Example 1: Uninstall TiDB v6.0.0. +Example 1: Uninstall TiDB v5.4.0. {{< copyable "shell-regular" >}} ```shell -tiup uninstall tidb:v6.0.0 +tiup uninstall tidb:v5.4.0 ``` Example 2: Uninstall TiKV of all versions. diff --git a/tiup/tiup-mirror.md b/tiup/tiup-mirror.md index 6e898ba8872ce..2b830d94a9a84 100644 --- a/tiup/tiup-mirror.md +++ b/tiup/tiup-mirror.md @@ -88,9 +88,9 @@ The `tiup mirror clone` command provides many optional flags (might provide more If you want to clone only one version (not all versions) of a component, use `--=` to specify this version. For example: - - Execute the `tiup mirror clone --tidb v6.0.0` command to clone the v6.0.0 version of the TiDB component. - - Run the `tiup mirror clone --tidb v6.0.0 --tikv all` command to clone the v6.0.0 version of the TiDB component and all versions of the TiKV component. - - Run the `tiup mirror clone v6.0.0` command to clone the v6.0.0 version of all components in a cluster. + - Execute the `tiup mirror clone --tidb v5.4.0` command to clone the v5.4.0 version of the TiDB component. + - Run the `tiup mirror clone --tidb v5.4.0 --tikv all` command to clone the v5.4.0 version of the TiDB component and all versions of the TiKV component. + - Run the `tiup mirror clone v5.4.0` command to clone the v5.4.0 version of all components in a cluster. After cloning, signing keys are set up automatically. diff --git a/tiup/tiup-playground.md b/tiup/tiup-playground.md index 3b72b4206b6c6..6d194428e2f9f 100644 --- a/tiup/tiup-playground.md +++ b/tiup/tiup-playground.md @@ -20,9 +20,9 @@ If you directly execute the `tiup playground` command, TiUP uses the locally ins This command actually performs the following operations: -- Because this command does not specify the version of the playground component, TiUP first checks the latest version of the installed playground component. Assume that the latest version is v1.9.3, then this command works the same as `tiup playground:v1.9.3`. +- Because this command does not specify the version of the playground component, TiUP first checks the latest version of the installed playground component. Assume that the latest version is v1.9.0, then this command works the same as `tiup playground:v1.9.0`. - If you have not used TiUP playground to install the TiDB, TiKV, and PD components, the playground component installs the latest stable version of these components, and then start these instances. -- Because this command does not specify the version of the TiDB, PD, and TiKV component, TiUP playground uses the latest version of each component by default. Assume that the latest version is v6.0.0, then this command works the same as `tiup playground:v1.9.3 v6.0.0`. +- Because this command does not specify the version of the TiDB, PD, and TiKV component, TiUP playground uses the latest version of each component by default. Assume that the latest version is v5.4.0, then this command works the same as `tiup playground:v1.9.0 v5.4.0`. - Because this command does not specify the number of each component, TiUP playground, by default, starts a smallest cluster that consists of one TiDB instance, one TiKV instance, one PD instance, and one TiFlash instance. - After starting each TiDB component, TiUP playground reminds you that the cluster is successfully started and provides you some useful information, such as how to connect to the TiDB cluster through the MySQL client and how to access the [TiDB Dashboard](/dashboard/dashboard-intro.md). diff --git a/upgrade-tidb-using-tiup.md b/upgrade-tidb-using-tiup.md index 653db1cd803e5..8bd0c4cfe401e 100644 --- a/upgrade-tidb-using-tiup.md +++ b/upgrade-tidb-using-tiup.md @@ -8,22 +8,25 @@ aliases: ['/docs/dev/upgrade-tidb-using-tiup/','/docs/dev/how-to/upgrade/using-t This document is targeted for the following upgrade paths: -- Upgrade from TiDB 4.0 versions to TiDB 6.0. -- Upgrade from TiDB 5.0-5.4 versions to TiDB 6.0. +- Upgrade from TiDB 4.0 versions to TiDB 5.4 versions. +- Upgrade from TiDB 5.0 versions to TiDB 5.4 versions. +- Upgrade from TiDB 5.1 versions to TiDB 5.4 versions. +- Upgrade from TiDB 5.2 versions to TiDB 5.4 versions. +- Upgrade from TiDB 5.3 versions to TiDB 5.4 versions. > **Note:** > -> If your cluster to be upgraded is v3.1 or an earlier version (v3.0 or v2.1), the direct upgrade to v6.0 or its patch versions is not supported. You need to upgrade your cluster first to v4.0 and then to v6.0. +> If your cluster to be upgraded is v3.1 or an earlier version (v3.0 or v2.1), the direct upgrade to v5.4 or its patch versions is not supported. You need to upgrade your cluster first to v4.0 and then to v5.4. ## Upgrade caveat - TiDB currently does not support version downgrade or rolling back to an earlier version after the upgrade. -- For the v4.0 cluster managed using TiDB Ansible, you need to import the cluster to TiUP (`tiup cluster`) for new management according to [Upgrade TiDB Using TiUP (v4.0)](https://docs.pingcap.com/tidb/v4.0/upgrade-tidb-using-tiup#import-tidb-ansible-and-the-inventoryini-configuration-to-tiup). Then you can upgrade the cluster to v6.0 or its patch versions according to this document. -- To update versions earlier than 3.0 to 6.0: +- For the v4.0 cluster managed using TiDB Ansible, you need to import the cluster to TiUP (`tiup cluster`) for new management according to [Upgrade TiDB Using TiUP (v4.0)](https://docs.pingcap.com/tidb/v4.0/upgrade-tidb-using-tiup#import-tidb-ansible-and-the-inventoryini-configuration-to-tiup). Then you can upgrade the cluster to v5.4 or its patch versions according to this document. +- To update versions earlier than 3.0 to 5.4: 1. Update this version to 3.0 using [TiDB Ansible](https://docs.pingcap.com/tidb/v3.0/upgrade-tidb-using-ansible). 2. Use TiUP (`tiup cluster`) to import the TiDB Ansible configuration. 3. Update the 3.0 version to 4.0 according to [Upgrade TiDB Using TiUP (v4.0)](https://docs.pingcap.com/tidb/v4.0/upgrade-tidb-using-tiup#import-tidb-ansible-and-the-inventoryini-configuration-to-tiup). - 4. Upgrade the cluster to v6.0 according to this document. + 4. Upgrade the cluster to v5.4 according to this document. - Support upgrading the versions of TiDB Binlog, TiCDC, TiFlash, and other components. - For detailed compatibility changes of different versions, see the [Release Notes](/releases/release-notes.md) of each version. Modify your cluster configuration according to the "Compatibility Changes" section of the corresponding release notes. @@ -47,7 +50,7 @@ Before upgrading your TiDB cluster, you first need to upgrade TiUP or TiUP mirro > > If the control machine of the cluster to upgrade cannot access `https://tiup-mirrors.pingcap.com`, skip this section and see [Upgrade TiUP offline mirror](#upgrade-tiup-offline-mirror). -1. Upgrade the TiUP version. It is recommended that the TiUP version is `1.9.3` or later. +1. Upgrade the TiUP version. It is recommended that the TiUP version is `1.9.0` or later. {{< copyable "shell-regular" >}} @@ -56,7 +59,7 @@ Before upgrading your TiDB cluster, you first need to upgrade TiUP or TiUP mirro tiup --version ``` -2. Upgrade the TiUP Cluster version. It is recommended that the TiUP Cluster version is `1.9.3` or later. +2. Upgrade the TiUP Cluster version. It is recommended that the TiUP Cluster version is `1.9.0` or later. {{< copyable "shell-regular" >}} @@ -98,7 +101,7 @@ Now, the offline mirror has been upgraded successfully. If an error occurs durin > Skip this step if one of the following situations applies: > > + You have not modified the configuration parameters of the original cluster. Or you have modified the configuration parameters using `tiup cluster` but no more modification is needed. -> + After the upgrade, you want to use v6.0's default parameter values for the unmodified configuration items. +> + After the upgrade, you want to use v5.4's default parameter values for the unmodified configuration items. 1. Enter the `vi` editing mode to edit the topology file: @@ -114,7 +117,7 @@ Now, the offline mirror has been upgraded successfully. If an error occurs durin > **Note:** > -> Before you upgrade the cluster to v6.0, make sure that the parameters you have modified in v4.0 are compatible in v6.0. For details, see [TiKV Configuration File](/tikv-configuration-file.md). +> Before you upgrade the cluster to v5.4, make sure that the parameters you have modified in v4.0 are compatible in v5.4. For details, see [TiKV Configuration File](/tikv-configuration-file.md). > > The following three TiKV parameters are obsolete in TiDB v5.0. If the following parameters have been configured in your original cluster, you need to delete these parameters through `edit-config`: > @@ -157,12 +160,12 @@ If your application has a maintenance window for the database to be stopped for tiup cluster upgrade ``` -For example, if you want to upgrade the cluster to v6.0.0: +For example, if you want to upgrade the cluster to v5.4.0: {{< copyable "shell-regular" >}} ```shell -tiup cluster upgrade v6.0.0 +tiup cluster upgrade v5.4.0 ``` > **Note:** @@ -212,7 +215,7 @@ tiup cluster display ``` Cluster type: tidb Cluster name: -Cluster version: v6.0.0 +Cluster version: v5.4.0 ``` > **Note:** @@ -262,10 +265,10 @@ You can upgrade the tool version by using TiUP to install the `ctl` component of {{< copyable "shell-regular" >}} ```shell -tiup install ctl:v6.0.0 +tiup install ctl:v5.4.0 ``` -## TiDB 6.0 compatibility changes +## TiDB 5.4 compatibility changes -- See TiDB 6.0 Release Notes for the compatibility changes. +- See TiDB 5.4 Release Notes for the compatibility changes. - Try to avoid creating a new clustered index table when you apply rolling updates to the clusters using TiDB Binlog. From 38d6d75bbae478a626c73e2d3dfff7335727a844 Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Sat, 2 Apr 2022 12:06:10 +0800 Subject: [PATCH 34/36] Revert "Merge branch 'master' into pr/8037" This reverts commit 4bef2ffb806c2328554c3c7ac0f09106df5410f1, reversing changes made to 0fe95644f2918403d4fe63713790d1b58a109391. --- TOC.md | 2 - _index.md | 2 +- br/backup-and-restore-faq.md | 6 -- br/backup-and-restore-tool.md | 13 +--- br/br-batch-create-table.md | 65 ------------------- overview.md | 2 +- quick-start-with-tidb.md | 2 +- .../sql-statement-alter-placement-policy.md | 5 -- tidb-scheduling.md | 14 ++-- 9 files changed, 11 insertions(+), 100 deletions(-) delete mode 100644 br/br-batch-create-table.md diff --git a/TOC.md b/TOC.md index 83dccb5bcccbe..b662aa456ca4b 100644 --- a/TOC.md +++ b/TOC.md @@ -72,7 +72,6 @@ - [Back up and Restore Data on Azure Blob Storage](/br/backup-and-restore-azblob.md) - BR Features - [Auto Tune](/br/br-auto-tune.md) - - [Batch Create Table](/br/br-batch-create-table.md) - [BR FAQ](/br/backup-and-restore-faq.md) - [Configure Time Zone](/configure-time-zone.md) - [Daily Checklist](/daily-check.md) @@ -204,7 +203,6 @@ - [External Storages](/br/backup-and-restore-storages.md) - BR Features - [Auto Tune](/br/br-auto-tune.md) - - [Batch Create Table](/br/br-batch-create-table.md) - [BR FAQ](/br/backup-and-restore-faq.md) - TiDB Binlog - [Overview](/tidb-binlog/tidb-binlog-overview.md) diff --git a/_index.md b/_index.md index 55d9035631407..ac79c1dc3a156 100644 --- a/_index.md +++ b/_index.md @@ -8,7 +8,7 @@ aliases: ['/docs/dev/'] [TiDB](https://github.com/pingcap/tidb) (/’taɪdiːbi:/, "Ti" stands for Titanium) is an open-source, distributed, NewSQL database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads. It is MySQL compatible and features horizontal scalability, strong consistency, and high availability. TiDB can be deployed on-premise or in-cloud. -Designed for the cloud, TiDB provides flexible scalability, reliability and security on the cloud platform. Users can elastically scale TiDB to meet the requirements of their changing workloads. [TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/v1.1/tidb-operator-overview) helps manage TiDB on Kubernetes and automates operating tasks, which makes TiDB easier to deploy on any cloud that provides managed Kubernetes. [TiDB Cloud](https://pingcap.com/tidb-cloud/), the fully-managed TiDB service, is the easiest, most economical, and most resilient way to unlock the full power of [TiDB in the cloud](https://docs.pingcap.com/tidbcloud/), allowing you to deploy and run TiDB clusters with just a few clicks. +Designed for the cloud, TiDB provides flexible scalability, reliability and security on the cloud platform. Users can elastically scale TiDB to meet the requirements of their changing workloads. [TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/v1.1/tidb-operator-overview) helps manage TiDB on Kubernetes and automates operating tasks, which makes TiDB easier to deploy on any cloud that provides managed Kubernetes. [TiDB Cloud](https://pingcap.com/products/tidbcloud) (Public Preview), the fully-managed TiDB service, is the easiest, most economical, and most resilient way to unlock the full power of [TiDB in the cloud](https://docs.pingcap.com/tidbcloud/beta), allowing you to deploy and run TiDB clusters with just a few clicks. diff --git a/br/backup-and-restore-faq.md b/br/backup-and-restore-faq.md index 2e6ece4643fc2..6dca31b9c41c4 100644 --- a/br/backup-and-restore-faq.md +++ b/br/backup-and-restore-faq.md @@ -164,12 +164,6 @@ You can use [`filter.rules`](https://github.com/pingcap/tiflow/blob/7c3c2336f981 Yes. BR backs up the [`SHARD_ROW_ID_BITS` and `PRE_SPLIT_REGIONS`](/sql-statements/sql-statement-split-region.md#pre_split_regions) information of a table. The data of the restored table is also split into multiple Regions. -## What should I do if the restore fails with the error message `the entry too large, the max entry size is 6291456, the size of data is 7690800`? - -You can try to reduce the number of tables to be created in a batch by setting `--ddl-batch-size` to `128` or a smaller value. - -When using BR to restore the backup data with the value of [`--ddl-batch-size`](/br/br-batch-create-table.md#how to use) greater than `1`, TiDB writes a DDL job of table creation to the DDL jobs queue that is maintained by TiKV. At this time, the total size of all tables schema sent by TiDB at one time should not exceed 6 MB, because the maximum value of job messages is `6 MB` by default (it is **not recommended** to modify this value. For details, see [`txn-entry-size-limit`](/tidb-configuration-file.md#txn-entry-size-limit-new-in-v50) and [`raft-entry-max-size`](/tikv-configuration-file.md#raft-entry-max-size)). Therefore, if you set `--ddl-batch-size` to an excessively large value, the schema size of the tables sent by TiDB in a batch at one time exceeds the specified value, which causes BR to report the `entry too large, the max entry size is 6291456, the size of data is 7690800` error. - ## Why is the `region is unavailable` error reported for a SQL query after I use BR to restore the backup data? If the cluster backed up using BR has TiFlash, `TableInfo` stores the TiFlash information when BR restores the backup data. If the cluster to be restored does not have TiFlash, the `region is unavailable` error is reported. diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 1892bdb221269..4ec4e973f1d67 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -156,21 +156,10 @@ Note that skipping the version check might introduce incompatibility. The versio | Backup version (vertical) \ Restore version (horizontal) | Use BR nightly to restore TiDB nightly | Use BR v5.0 to restore TiDB v5.0| Use BR v4.0 to restore TiDB v4.0 | | ---- | ---- | ---- | ---- | | Use BR nightly to back up TiDB nightly | ✅ | ✅ | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | -| Use BR v5.0 to back up TiDB v5.0 | ✅ | ✅ | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) +| Use BR v5.0 to back up TiDB v5.0 | ✅ | ✅ | ❌ (If a table with the primary key of the non-integer clustered index type is restored to a TiDB v4.0 cluster, BR will cause data error without warning.) | Use BR v4.0 to back up TiDB v4.0 | ✅ | ✅ | ✅ (If TiKV >= v4.0.0-rc.1, and if BR contains the [#233](https://github.com/pingcap/br/pull/233) bug fix and TiKV does not contain the [#7241](https://github.com/tikv/tikv/pull/7241) bug fix, BR will cause the TiKV node to restart.) | | Use BR nightly or v5.0 to back up TiDB v4.0 | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | ❌ (If the TiDB version is earlier than v4.0.9, the [#609](https://github.com/pingcap/br/issues/609) issue might occur.) | -#### Check for the `new_collations_enabled_on_first_bootstrap` variable - -Since TiDB v6.0.0, the default value of [`new_collations_enabled_on_first_bootstrap`](/tidb-configuration-file.md#new_collations_enabled_on_first_bootstrap) has changed from `false` to `true`. When the value of `new_collations_enabled_on_first_bootstrap` is consistent between the upstream and downstream clusters, BR safely restores the data backed up in the upstream cluster to the downstream cluster. - -Since v6.0.0, BR backs up the `new_collations_enabled_on_first_bootstrap` configuration of the upstream cluster and then checks whether the value of this configuration is consistent between the upstream and downstream clusters. If the value is inconsistent between the upstream and downstream clusters, BR does not perform the data restore and reports an error. - -Suppose that you have backed up the data in a TiDB cluster of an earlier version of v6.0.0, and you want to restore this data to a TiDB cluster of v6.0.0 or later versions. In this situation, you need manually to check whether the value of `new_collations_enabled_on_first_bootstrap` is consistent between the upstream and downstream clusters: - -- If the value is consistent, you can add `--check-requirements=false` to the restore command to skip this configuration check. -- If the value is inconsistent, and you forcibly perform the restore, BR reports a [data validation error](/br/backup-and-restore-tool.md#usage-restrictions). - ### Back up and restore table data in the `mysql` system schema (experimental feature) > **Warning:** diff --git a/br/br-batch-create-table.md b/br/br-batch-create-table.md deleted file mode 100644 index b10e066f6185c..0000000000000 --- a/br/br-batch-create-table.md +++ /dev/null @@ -1,65 +0,0 @@ ---- -title: Batch Create Table -summary: Learn how to use the Batch Create Table feature. When restoring data, BR can create tables in batches to speed up the restore process. ---- - -# Batch Create Table - -When restoring data, Backup & Restore (BR) creates databases and tables in the target TiDB cluster and then restores the backup data to the tables. In versions earlier than TiDB v6.0.0, BR uses the [serial execution](#implementation-principles) implementation to create tables in the restore process. However, when BR restores data with a large number of tables (nearly 50000), this implementation takes much time on creating tables. - -To speed up the table creation process and reduce the time for restoring data, the Batch Create Table feature is introduced in TiDB v6.0.0. This feature is enabled by default. - -> **Note:** -> -> - To use the Batch Create Table feature, both TiDB and BR are expected to be of v6.0.0 or later. If either TiDB or BR is earlier than v6.0.0, BR uses the serial execution implementation. -> - Suppose that you use a cluster management tool (for example, TiUP), and your TiDB and BR are of v6.0.0 or later versions, or your TiDB and BR are upgraded from a version earlier than v6.0.0 to v6.0.0 or later. In this case, BR enables the Batch Create Table feature by default. - -## Usage scenario - -If you need to restore data with a massive amount of tables, for example, 50000 tables, you can use the Batch Create Table feature to speed up the restore process. - -For the detailed effect, see [Test for the Batch Create Table Feature](#test-for-the-batch-create-table-feature). - -## Use the Batch Create Table feature - -BR enables the Batch Create Table feature by default, with the default configuration of `--ddl-batch-size=128` in v6.0.0 or later to speed up the restore process. Therefore, you do not need to configure this parameter. `--ddl-batch-size=128` means that BR creates tables in batches, each batch with 128 tables. - -To disable this feature, you can set `--ddl-batch-size` to `0`. See the following example command: - -{{< copyable "shell-regular" >}} - -```shell -br restore full -s local:///br_data/ --pd 172.16.5.198:2379 --log-file restore.log --ddl-batch-size=0 -``` - -After this feature is disabled, BR uses the [serial execution implementation](#implementation-principles) instead. - -## Implementation principles - -- Serial execution implementation before v6.0.0: - - When restoring data, BR creates databases and tables in the target TiDB cluster and then restores the backup data to the tables. To create tables, BR calls TiDB internal API first, and then processes table creation tasks, which works similarly to executing the `Create Table` statement by BR. The TiDB DDL owner creates tables sequentially. Once the DDL owner creates a table, the DDL schema version changes correspondingly and each version change is synchronized to other TiDB DDL workers (including BR). Therefore, when BR restores a large number of tables, the serial execution implementation is time-consuming. - -- Batch create table implementation since v6.0.0: - - By default, BR creates tables in multiple batches, and each batch has 128 tables. Using this implementation, when BR creates one batch of tables, the TiDB schema version only changes once. This implementation significantly increases the speed of table creation. - -## Test for the Batch Create Table feature - -This section describes the test information about the Batch Create Table feature. The test environment is as follows: - -- Cluster configurations: - - - 15 TiKV instances. Each TiKV instance is equipped with 16 CPU cores, 80 GB memory, and 16 threads to process RPC requests ([`import.num-threads`](/tikv-configuration-file.md#num-threads) = 16). - - 3 TiDB instances. Each TiDB instance is equipped with 16 CPU cores, 32 GB memory. - - 3 PD instances. Each PD instance is equipped with 16 CPU cores, 32 GB memory. - -- The size of data to be restored: 16.16 TB - -The test result is as follows: - -``` -‘[2022/03/12 22:37:49.060 +08:00] [INFO] [collector.go:67] ["Full restore success summary"] [total-ranges=751760] [ranges-succeed=751760] [ranges-failed=0] [split-region=1h33m18.078448449s] [restore-ranges=542693] [total-take=1h41m35.471476438s] [restore-data-size(after-compressed)=8.337TB] [Size=8336694965072] [BackupTS=431773933856882690] [total-kv=148015861383] [total-kv-size=16.16TB] [average-speed=2.661GB/s]’ -``` - -From the test result, you can see that the average speed of restoring one TiKV instance is as high as 181.65 MB/s (which equals to `average-speed`/`tikv_count`）. \ No newline at end of file diff --git a/overview.md b/overview.md index c9532807c16fd..772d6fdd6918e 100644 --- a/overview.md +++ b/overview.md @@ -26,7 +26,7 @@ aliases: ['/docs/dev/key-features/','/tidb/dev/key-features','/docs/dev/overview - **Cloud-native distributed database** - TiDB is a distributed database designed for the cloud, providing flexible scalability, reliability and security on the cloud platform. Users can elastically scale TiDB to meet the requirements of their changing workloads. In TiDB, each piece of data has 3 replicas at least, which can be scheduled in different cloud availability zones to tolerate the outage of a whole data center. [TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/stable/tidb-operator-overview) helps manage TiDB on Kubernetes and automates tasks related to operating the TiDB cluster, which makes TiDB easier to deploy on any cloud that provides managed Kubernetes. [TiDB Cloud](https://pingcap.com/tidb-cloud/), the fully-managed TiDB service, is the easiest, most economical, and most resilient way to unlock the full power of [TiDB in the cloud](https://docs.pingcap.com/tidbcloud/), allowing you to deploy and run TiDB clusters with just a few clicks. + TiDB is a distributed database designed for the cloud, providing flexible scalability, reliability and security on the cloud platform. Users can elastically scale TiDB to meet the requirements of their changing workloads. In TiDB, each piece of data has 3 replicas at least, which can be scheduled in different cloud availability zones to tolerate the outage of a whole data center. [TiDB Operator](https://docs.pingcap.com/tidb-in-kubernetes/stable/tidb-operator-overview) helps manage TiDB on Kubernetes and automates tasks related to operating the TiDB cluster, which makes TiDB easier to deploy on any cloud that provides managed Kubernetes. [TiDB Cloud](https://pingcap.com/products/tidbcloud) (Beta), the fully-managed TiDB service, is the easiest, most economical, and most resilient way to unlock the full power of [TiDB in the cloud](https://docs.pingcap.com/tidbcloud/beta), allowing you to deploy and run TiDB clusters with just a few clicks. - **Compatible with the MySQL 5.7 protocol and MySQL ecosystem** diff --git a/quick-start-with-tidb.md b/quick-start-with-tidb.md index 7d090bc96fc03..e00f8ae14fcb2 100644 --- a/quick-start-with-tidb.md +++ b/quick-start-with-tidb.md @@ -19,7 +19,7 @@ This guide walks you through the quickest way to get started with TiDB. For non- > > - To deploy an on-premises production cluster, see [production installation guide](/production-deployment-using-tiup.md). > - To deploy TiDB in Kubernetes, see [Get Started with TiDB in Kubernetes](https://docs.pingcap.com/tidb-in-kubernetes/stable/get-started). -> - To manage TiDB in the cloud, see [TiDB Cloud Quick Start](https://docs.pingcap.com/tidbcloud/tidb-cloud-quickstart). +> - To manage TiDB in the cloud, see [TiDB Cloud Quick Start](https://docs.pingcap.com/tidbcloud/beta/tidb-cloud-quickstart). ## Deploy a local test cluster diff --git a/sql-statements/sql-statement-alter-placement-policy.md b/sql-statements/sql-statement-alter-placement-policy.md index f3744ad2364c7..9a13e03798388 100644 --- a/sql-statements/sql-statement-alter-placement-policy.md +++ b/sql-statements/sql-statement-alter-placement-policy.md @@ -9,11 +9,6 @@ summary: The usage of ALTER PLACEMENT POLICY in TiDB. `ALTER PLACEMENT POLICY` _replaces_ the previous policy with the new definition. It does not _merge_ the old policy with the new one. In the following example, `FOLLOWERS=4` is lost when the `ALTER PLACEMENT POLICY` is executed: -```sql -CREATE PLACEMENT POLICY p1 FOLLOWERS=4; -ALTER PLACEMENT POLICY p1 PRIMARY_REGION="us-east-1" REGIONS="us-east-1,us-west-1"; -``` - ## Synopsis ```ebnf+diagram diff --git a/tidb-scheduling.md b/tidb-scheduling.md index 4beb76c372467..564bb2d3247f3 100644 --- a/tidb-scheduling.md +++ b/tidb-scheduling.md @@ -77,15 +77,15 @@ Scheduling is based on information collection. In short, the PD scheduling compo * Whether the store is overloaded * Labels (See [Perception of Topology](/schedule-replicas-by-topology-labels.md)) - You can use PD control to check the status of a TiKV store, which can be Up, Disconnect, Offline, Down, or Tombstone. The following is a description of all statuses and their relationship. +You can use PD control to check the status of a TiKV store, which can be Up, Disconnect, Offline, Down, or Tombstone. The following is a description of all statuses and their relationship. - + **Up**: The TiKV store is in service. - + **Disconnect**: Heartbeat messages between the PD and the TiKV store are lost for more than 20 seconds. If the lost period exceeds the time specified by `max-store-down-time`, the status "Disconnect" changes to "Down". - + **Down**: Heartbeat messages between the PD and the TiKV store are lost for a time longer than `max-store-down-time` (30 minutes by default). In this status, the TiKV store starts replenishing replicas of each Region on the surviving store. - + **Offline**: A TiKV store is manually taken offline through PD Control. This is only an intermediate status for the store to go offline. The store in this status moves all its Regions to other "Up" stores that meet the relocation conditions. When `leader_count` and `region_count` (obtained through PD Control) both show `0`, the store status changes to "Tombstone" from "Offline". In the "Offline" status, **do not** disable the store service or the physical server where the store is located. During the process that the store goes offline, if the cluster does not have target stores to relocate the Regions (for example, inadequate stores to hold replicas in the cluster), the store is always in the "Offline" status. - + **Tombstone**: The TiKV store is completely offline. You can use the `remove-tombstone` interface to safely clean up TiKV in this status. ++ **Up**: The TiKV store is in service. ++ **Disconnect**: Heartbeat messages between the PD and the TiKV store are lost for more than 20 seconds. If the lost period exceeds the time specified by `max-store-down-time`, the status "Disconnect" changes to "Down". ++ **Down**: Heartbeat messages between the PD and the TiKV store are lost for a time longer than `max-store-down-time` (30 minutes by default). In this status, the TiKV store starts replenishing replicas of each Region on the surviving store. ++ **Offline**: A TiKV store is manually taken offline through PD Control. This is only an intermediate status for the store to go offline. The store in this status moves all its Regions to other "Up" stores that meet the relocation conditions. When `leader_count` and `region_count` (obtained through PD Control) both show `0`, the store status changes to "Tombstone" from "Offline". In the "Offline" status, **do not** disable the store service or the physical server where the store is located. During the process that the store goes offline, if the cluster does not have target stores to relocate the Regions (for example, inadequate stores to hold replicas in the cluster), the store is always in the "Offline" status. ++ **Tombstone**: The TiKV store is completely offline. You can use the `remove-tombstone` interface to safely clean up TiKV in this status. - ![TiKV store status relationship](/media/tikv-store-status-relationship.png) +![TiKV store status relationship](/media/tikv-store-status-relationship.png) - Information reported by Region leaders: From 06f60df843059ac764862d8f2363f37cfef1033a Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Fri, 1 Apr 2022 20:16:29 +0800 Subject: [PATCH 35/36] This is an automated cherry-pick of #7975 Signed-off-by: ti-chi-bot --- sql-plan-management.md | 321 ++++++++++++++++++++++++++++++----------- system-variables.md | 9 ++ 2 files changed, 249 insertions(+), 81 deletions(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 2c1d4fa039643..9891fc73b4f04 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -21,42 +21,46 @@ CREATE [GLOBAL | SESSION] BINDING FOR BindableStmt USING BindableStmt This statement binds SQL execution plans at the GLOBAL or SESSION level. Currently, supported bindable SQL statements (BindableStmt) in TiDB include `SELECT`, `DELETE`, `UPDATE`, and `INSERT` / `REPLACE` with `SELECT` subqueries. +> **Note:** +> +> Bindings have higher priority over manually added hints. Therefore, when you execute a statement containing a hint while a corresponding binding is present, the hint controlling the behavior of the optimizer does not take effect. However, other types of hints are still effective. + Specifically, two types of these statements cannot be bound to execution plans due to syntax conflicts. See the following examples: ```sql --- Type one: Statements that get the Cartesian product by using the `join` keyword and not specifying the associated columns with the `using` keyword. -create global binding for - select * from t t1 join t t2 -using - select * from t t1 join t t2; - --- Type two: `DELETE` statements that contain the `using` keyword. -create global binding for - delete from t1 using t1 join t2 on t1.a = t2.a -using - delete from t1 using t1 join t2 on t1.a = t2.a; +-- Type one: Statements that get the Cartesian product by using the `JOIN` keyword and not specifying the associated columns with the `USING` keyword. +CREATE GLOBAL BINDING for + SELECT * FROM t t1 JOIN t t2 +USING + SELECT * FROM t t1 JOIN t t2; + +-- Type two: `DELETE` statements that contain the `USING` keyword. +CREATE GLOBAL BINDING for + DELETE FROM t1 USING t1 JOIN t2 ON t1.a = t2.a +USING + DELETE FROM t1 USING t1 JOIN t2 ON t1.a = t2.a; ``` You can bypass syntax conflicts by using equivalent statements. For example, you can rewrite the above statements in the following ways: ```sql --- First rewrite of type one statements: Add a `using` clause for the `join` keyword. -create global binding for - select * from t t1 join t t2 using (a) +-- First rewrite of type one statements: Add a `USING` clause for the `JOIN` keyword. +CREATE GLOBAL BINDING for + SELECT * FROM t t1 JOIN t t2 USING (a) +USING + SELECT * FROM t t1 JOIN t t2 USING (a); + +-- Second rewrite of type one statements: Delete the `JOIN` keyword. +CREATE GLOBAL BINDING for + SELECT * FROM t t1, t t2 +USING + SELECT * FROM t t1, t t2; + +-- Rewrite of type two statements: Remove the `USING` keyword from the `delete` statement. +CREATE GLOBAL BINDING for + DELETE t1 FROM t1 JOIN t2 ON t1.a = t2.a using - select * from t t1 join t t2 using (a); - --- Second rewrite of type one statements: Delete the `join` keyword. -create global binding for - select * from t t1, t t2 -using - select * from t t1, t t2; - --- Rewrite of type two statements: Remove the `using` keyword from the `delete` statement. -create global binding for - delete t1 from t1 join t2 on t1.a = t2.a -using - delete t1 from t1 join t2 on t1.a = t2.a; + DELETE t1 FROM t1 JOIN t2 ON t1.a = t2.a; ``` > **Note:** @@ -67,16 +71,16 @@ Here are two examples: ```sql -- The hint takes effect in the following statement. -create global binding for - insert into t1 select * from t2 where a > 1 and b = 1 +CREATE GLOBAL BINDING for + INSERT INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1 using - insert into t1 select /*+ use_index(@sel_1 t2, a) */ * from t2 where a > 1 and b = 1; + INSERT INTO t1 SELECT /*+ use_index(@sel_1 t2, a) */ * FROM t2 WHERE a > 1 AND b = 1; -- The hint cannot take effect in the following statement. -create global binding for - insert into t1 select * from t2 where a > 1 and b = 1 +CREATE GLOBAL BINDING for + INSERT INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1 using - insert /*+ use_index(@sel_1 t2, a) */ into t1 select * from t2 where a > 1 and b = 1; + INSERT /*+ use_index(@sel_1 t2, a) */ INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1; ``` If you do not specify the scope when creating an execution plan binding, the default scope is SESSION. The TiDB optimizer normalizes bound SQL statements and stores them in the system table. When processing SQL queries, if a normalized statement matches one of the bound SQL statements in the system table and the system variable `tidb_use_plan_baselines` is set to `on` (the default value is `on`), TiDB then uses the corresponding optimizer hint for this statement. If there are multiple matchable execution plans, the optimizer chooses the least costly one to bind. @@ -84,9 +88,9 @@ If you do not specify the scope when creating an execution plan binding, the def `Normalization` is a process that converts a constant in an SQL statement to a variable parameter and explicitly specifies the database for tables referenced in the query, with standardized processing on the spaces and line breaks in the SQL statement. See the following example: ```sql -select * from t where a > 1 +SELECT * FROM t WHERE a > 1 -- After normalization, the above statement is as follows: -select * from test . t where a > ? +SELECT * FROM test . t WHERE a > ? ``` > **Note:** @@ -96,15 +100,15 @@ select * from test . t where a > ? > For example: > > ```sql -> select * from t limit 10 -> select * from t limit 10, 20 -> select * from t where a in (1) -> select * from t where a in (1,2,3) +> SELECT * FROM t limit 10 +> SELECT * FROM t limit 10, 20 +> SELECT * FROM t WHERE a IN (1) +> SELECT * FROM t WHERE a IN (1,2,3) > -- After normalization, the above statements are as follows: -> select * from test . t limit ? -> select * from test . t limit ... -> select * from test . t where a in ( ? ) -> select * from test . t where a in ( ... ) +> SELECT * FROM test . t limit ? +> SELECT * FROM test . t limit ... +> SELECT * FROM test . t WHERE a IN ( ? ) +> SELECT * FROM test . t WHERE a IN ( ... ) > ``` > > When bindings are created, TiDB treats SQL statements that contain a single constant and SQL statements that contain multiple constants joined by commas differently. Therefore, you need to create bindings for the two SQL types separately. @@ -115,25 +119,25 @@ For example: ```sql -- Creates a GLOBAL binding and specifies using `sort merge join` in this binding. -create global binding for - select * from t1, t2 where t1.id = t2.id -using - select /*+ merge_join(t1, t2) */ * from t1, t2 where t1.id = t2.id; +CREATE GLOBAL BINDING for + SELECT * FROM t1, t2 WHERE t1.id = t2.id +USING + SELECT /*+ merge_join(t1, t2) */ * FROM t1, t2 WHERE t1.id = t2.id; -- The execution plan of this SQL statement uses the `sort merge join` specified in the GLOBAL binding. -explain select * from t1, t2 where t1.id = t2.id; +explain SELECT * FROM t1, t2 WHERE t1.id = t2.id; -- Creates another SESSION binding and specifies using `hash join` in this binding. -create binding for - select * from t1, t2 where t1.id = t2.id -using - select /*+ hash_join(t1, t2) */ * from t1, t2 where t1.id = t2.id; +CREATE BINDING for + SELECT * FROM t1, t2 WHERE t1.id = t2.id +USING + SELECT /*+ hash_join(t1, t2) */ * FROM t1, t2 WHERE t1.id = t2.id; -- In the execution plan of this statement, `hash join` specified in the SESSION binding is used, instead of `sort merge join` specified in the GLOBAL binding. -explain select * from t1, t2 where t1.id = t2.id; +explain SELECT * FROM t1, t2 WHERE t1.id = t2.id; ``` -When the first `select` statement is being executed, the optimizer adds the `sm_join(t1, t2)` hint to the statement through the binding in the GLOBAL scope. The top node of the execution plan in the `explain` result is MergeJoin. When the second `select` statement is being executed, the optimizer uses the binding in the SESSION scope instead of the binding in the GLOBAL scope and adds the `hash_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is HashJoin. +When the first `SELECT` statement is being executed, the optimizer adds the `sm_join(t1, t2)` hint to the statement through the binding in the GLOBAL scope. The top node of the execution plan in the `explain` result is MergeJoin. When the second `SELECT` statement is being executed, the optimizer uses the binding in the SESSION scope instead of the binding in the GLOBAL scope and adds the `hash_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is HashJoin. Each standardized SQL statement can have only one binding created using `CREATE BINDING` at a time. When multiple bindings are created for the same standardized SQL statement, the last created binding is retained, and all previous bindings (created and evolved) are marked as deleted. But session bindings and global bindings can coexist and are not affected by this logic. @@ -141,13 +145,13 @@ In addition, when you create a binding, TiDB requires that the session is in a d The original SQL statement and the bound statement must have the same text after normalization and hint removal, or the binding will fail. Take the following examples: -- This binding can be created successfully because the texts before and after parameterization and hint removal are the same: `select * from test . t where a > ?` +- This binding can be created successfully because the texts before and after parameterization and hint removal are the same: `SELECT * FROM test . t WHERE a > ?` ```sql CREATE BINDING FOR SELECT * FROM t WHERE a > 1 USING SELECT * FROM t use index (idx) WHERE a > 2 ``` -- This binding will fail because the original SQL statement is processed as `select * from test . t where a > ?`, while the bound SQL statement is processed differently as `select * from test . t where b > ?`. +- This binding will fail because the original SQL statement is processed as `SELECT * FROM test . t WHERE a > ?`, while the bound SQL statement is processed differently as `SELECT * FROM test . t WHERE b > ?`. ```sql CREATE BINDING FOR SELECT * FROM t WHERE a > 1 USING SELECT * FROM t use index(idx) WHERE b > 2 @@ -173,10 +177,10 @@ The following example is based on the example in [create binding](#create-a-bind ```sql -- Drops the binding created in the SESSION scope. -drop session binding for select * from t1, t2 where t1.id = t2.id; +drop session binding for SELECT * FROM t1, t2 WHERE t1.id = t2.id; -- Views the SQL execution plan again. -explain select * from t1,t2 where t1.id = t2.id; +explain SELECT * FROM t1,t2 WHERE t1.id = t2.id; ``` In the example above, the dropped binding in the SESSION scope shields the corresponding binding in the GLOBAL scope. The optimizer does not add the `sm_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is not fixed to MergeJoin by this hint. Instead, the top node is independently selected by the optimizer according to the cost estimation. @@ -185,7 +189,19 @@ In the example above, the dropped binding in the SESSION scope shields the corre > > Executing `DROP GLOBAL BINDING` drops the binding in the current tidb-server instance cache and changes the status of the corresponding row in the system table to 'deleted'. This statement does not directly delete the records in the system table, because other tidb-server instances need to read the 'deleted' status to drop the corresponding binding in their cache. For the records in these system tables with the status of 'deleted', at every 100 `bind-info-lease` (the default value is `3s`, and `300s` in total) interval, the background thread triggers an operation of reclaiming and clearing on the bindings of `update_time` before 10 `bind-info-lease` (to ensure that all tidb-server instances have read the 'deleted' status and updated the cache). -### View binding +## Change binding status + +{{< copyable "sql" >}} + +```sql +SET BINDING [ENABLED | DISABLED] FOR BindableStmt; +``` + +You can execute this statement to change the status of a binding. The default status is ENABLED. The effective scope is GLOBAL by default and cannot be modified. + +When executing this statement, you can only change the status of a binding from `Disabled` to `Enabled` or from `Enabled` to `Disabled`. If no binding is available for status changes, a warning message is returned, saying `There are no bindings can be set the status. Please check the SQL text`. Note that a binding in `Disabled` status is not used by any query. + +### View bindings {{< copyable "sql" >}} @@ -200,41 +216,98 @@ This statement outputs the execution plan bindings at the GLOBAL or SESSION leve | original_sql | Original SQL statement after parameterization | | bind_sql | Bound SQL statement with hints | | default_db | Default database | -| status | Status including Using, Deleted, Invalid, Rejected, and Pending verification| +| status | Status including Enabled (replacing the Using status from v6.0), Disabled, Deleted, Invalid, Rejected, and Pending verification| | create_time | Creating time | | update_time | Updating time | | charset | Character set | | collation | Ordering rule | | source | The way in which a binding is created, including `manual` (created by the `create [global] binding` SQL statement), `capture` (captured automatically by TiDB), and `evolve` (evolved automatically by TiDB) | -### Troubleshoot binding +### Troubleshoot a binding -{{< copyable "sql" >}} +You can use either of the following methods to troubleshoot a binding: -```sql -SELECT @@[SESSION.]last_plan_from_binding; -``` +- Use the system variable [`last_plan_from_binding`](/system-variables.md#last_plan_from_binding-new-in-v40) to show whether the execution plan used by the last executed statement is from the binding. -This statement uses the system variable [`last_plan_from_binding`](/system-variables.md#last_plan_from_binding-new-in-v40) to show whether the execution plan used by the last executed statement is from the binding. + {{< copyable "sql" >}} -In addition, when you use the `explain format = 'verbose'` statement to view the query plan of a SQL statement, if the SQL statement uses binding, the `explain` statement will return a warning. In this situation, you can check the warning message to learn which binding is used in the SQL statement. + ```sql + -- Create a global binding + CREATE GLOBAL BINDING for + SELECT * FROM t + USING + SELECT /*+ USE_INDEX(t, idx_a) */ * FROM t; + + SELECT * FROM t; + SELECT @@[SESSION.]last_plan_from_binding; + ``` -```sql --- Create a global binding. + ```sql + +--------------------------+ + | @@last_plan_from_binding | + +--------------------------+ + | 1 | + +--------------------------+ + 1 row in set (0.00 sec) + ``` -create global binding for - select * from t -using - select * from t; +- Use the `explain format = 'verbose'` statement to view the query plan of a SQL statement. If the SQL statement uses a binding, you can run `show warnings` to check which binding is used in the SQL statement. + + ```sql + -- Create a global binding + + CREATE GLOBAL BINDING for + SELECT * FROM t + USING + SELECT /*+ USE_INDEX(t, idx_a) */ * FROM t; + + -- Use explain format = 'verbose' to view the execution plan of a SQL statement + + explain format = 'verbose' SELECT * FROM t; + + -- Run `show warnings` to view the binding used in the query. + + show warnings; + ``` + + ```sql + +-------+------+--------------------------------------------------------------------------+ + | Level | Code | Message | + +-------+------+--------------------------------------------------------------------------+ + | Note | 1105 | Using the bindSQL: SELECT /*+ USE_INDEX(`t` `idx_a`)*/ * FROM `test`.`t` | + +-------+------+--------------------------------------------------------------------------+ + 1 row in set (0.01 sec) + + ``` + +### Cache bindings + +Each TiDB instance has a least recently used (LRU) cache for bindings. The cache capacity is controlled by the system variable [`tidb_mem_quota_binding_cache`](/system-variables.md#tidb_mem_quota_binding_cache-new-in-v600). You can view bindings that are cached in the TiDB instance. --- Use the `explain format = 'verbose'` statement to check the SQL execution plan. Check the warning message to view the binding used in the query. +To view the cache status of bindings, run the `SHOW binding_cache status` statement. In this statement, the effective scope is GLOBAL by default and cannot be modified. This statement returns the number of available bindings in the cache, the total number of available bindings in the system, memory usage of all cached bindings, and the total memory for the cache. -explain format = 'verbose' select * from t; -show warnings; +{{< copyable "sql" >}} + +```sql + +SHOW binding_cache status; +``` + +```sql ++-------------------+-------------------+--------------+--------------+ +| bindings_in_cache | bindings_in_table | memory_usage | memory_quota | ++-------------------+-------------------+--------------+--------------+ +| 1 | 1 | 159 Bytes | 64 MB | ++-------------------+-------------------+--------------+--------------+ +1 row in set (0.00 sec) ``` ## Baseline capturing +Used for [preventing regression of execution plans during an upgrade](#prevent-regression-of-execution-plans-during-an-upgrade), this feature captures queries that meet capturing conditions and creates bindings for these queries. + +### Enable capturing + To enable baseline capturing, set `tidb_capture_plan_baselines` to `on`. The default value is `off`. > **Note:** @@ -247,7 +320,12 @@ However, TiDB does not automatically capture bindings for the following types of - `EXPLAIN` and `EXPLAIN ANALYZE` statements. - SQL statements executed internally in TiDB, such as `SELECT` queries used for automatically loading statistical information. -- SQL statements that are bound to a manually created execution plan. +- Statements that contain `Enabled` or `Disabled` bindings. +- Statements that are filtered out by capturing conditions. + +> **Note:** +> +> Currently, a binding generates a group of hints to fix an execution plan generated by a query statement. In this way, for the same query, the execution plan does not change. For most OLTP queries, including queries using the same index or Join algorithm (such as HashJoin and IndexJoin), TiDB guarantees plan consistency before and after the binding. However, due to the limitations of hints, TiDB cannot guarantee plan consistency for some complex queries, such as Join of more than two tables, MPP queries, and complex OLAP queries. For `PREPARE` / `EXECUTE` statements and for queries executed with binary protocols, TiDB automatically captures bindings for the real query statements, not for the `PREPARE` / `EXECUTE` statements. @@ -255,6 +333,75 @@ For `PREPARE` / `EXECUTE` statements and for queries executed with binary protoc > > Because TiDB has some embedded SQL statements to ensure the correctness of some features, baseline capturing by default automatically shields these SQL statements. +### Filter out bindings + +This feature allows you to configure a blocklist to filter out queries whose bindings you do not want to capture. A blocklist has three dimensions, table name, frequency, and user name. + +#### Usage + +Insert filtering conditions into the system table `mysql.capture_plan_baselines_blacklist`. Then the filtering conditions take effect in the entire cluster immediately. + +```sql +-- Filter by table name + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'test.t'); + +-- Filter by database name and table name through wildcards + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'test.table_*'); + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'db_*.table_*'); + +-- Filter by frequency + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('frequency', '2'); + +-- Filter by user name + INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('user', 'user1'); +``` + +| **Dimension name** | **Description** | Remarks | +| :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | +| table | Filter by table name. Each filtering rule is in the `db.table` format. The supported filtering syntax includes [Plain table names](/table-filter.md#plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If a table name contains illegal characters, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | +| frequency | Filter by frequency. SQL statements executed more than once are captured by default. You can set a high frequency to capture statements that are frequently executed. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | +| user | Filter by user name. Statements executed by blocklisted users are not captured. | If multiple users execute the same statement and their user names are all in the blocklist, this statement is not captured. | + +> **Note:** +> +> - Modifying a blocklist requires the super privilege. +> +> - If a blocklist contains illegal filters, TiDB returns the warning message `[sql-bind] unknown capture filter type, ignore it` in the log. + +### Prevent regression of execution plans during an upgrade + + Before upgrading a TiDB cluster, you can use baseline capturing to prevent regression of execution plans by performing the following steps: + +1. Enable baseline capturing and keep it working for a period of time. + + > **Note:** + > + > Test data shows that long-term working of baseline capturing has a slight impact on the performance of the cluster load. Therefore, it is recommended to enable baseline capturing as long as possible so that important plans (appear twice or above) are captured. + +2. Upgrade the TiDB cluster. After the upgrade, TiDB uses those captured bindings to ensure execution plan consistency. + +3. After the upgrade, delete bindings as required. + + - Check the binding source by running the [`SHOW GLOBAL BINDINGS`](#view-bindings) statement. + + In the output, check the `Source` field to see whether a binding is captured (`capture`) or manually created (`manual`). + + - Determine whether to retain the captured bindings: + + ``` + -- View the plan with the binding enabled + SET @@SESSION.TIDB_USE_PLAN_BASELINES = true; + EXPLAIN FORMAT='VERBOSE' SELECT * FROM t1 WHERE ...; + + -- View the plan with the binding disabled + SET @@SESSION.TIDB_USE_PLAN_BASELINES = false; + EXPLAIN FORMAT='VERBOSE' SELECT * FROM t1 WHERE ...; + ``` + + - If the execution plan is consistent, you can delete the binding safely. + + - If the execution plan is inconsistent, you need to identify the cause, for example, by checking statistics. In this case, you need to retain the binding to ensure plan consistency. + ## Baseline evolution Baseline evolution is an important feature of SPM introduced in TiDB v4.0. @@ -270,7 +417,7 @@ Use the following statement to enable automatic binding evolution: {{< copyable "sql" >}} ```sql -set global tidb_evolve_plan_baselines = on; +SET GLOBAL tidb_evolve_plan_baselines = ON; ``` The default value of `tidb_evolve_plan_baselines` is `off`. @@ -287,7 +434,7 @@ Assume that table `t` is defined as follows: {{< copyable "sql" >}} ```sql -create table t(a int, b int, key(a), key(b)); +CREATE TABLE t(a INT, b INT, KEY(a), KEY(b)); ``` Perform the following query on table `t`: @@ -295,7 +442,7 @@ Perform the following query on table `t`: {{< copyable "sql" >}} ```sql -select * from t where a < 100 and b < 100; +SELECT * FROM t WHERE a < 100 AND b < 100; ``` In the table defined above, few rows meet the `a < 100` condition. But for some reason, the optimizer mistakenly selects the full table scan instead of the optimal execution plan that uses index `a`. You can first use the following statement to create a binding: @@ -303,7 +450,7 @@ In the table defined above, few rows meet the `a < 100` condition. But for some {{< copyable "sql" >}} ```sql -create global binding for select * from t where a < 100 and b < 100 using select * from t use index(a) where a < 100 and b < 100; +CREATE GLOBAL BINDING for SELECT * FROM t WHERE a < 100 AND b < 100 using SELECT * FROM t use index(a) WHERE a < 100 AND b < 100; ``` When the query above is executed again, the optimizer selects index `a` (influenced by the binding created above) to reduce the query time. @@ -349,11 +496,19 @@ During cluster upgrade, SQL Plan Management (SPM) might cause compatibility issu ```sql -- Check whether `tidb_evolve_plan_baselines` is disabled in the earlier version. +<<<<<<< HEAD select @@global.tidb_evolve_plan_baselines; -- If `tidb_evolve_plan_baselines` is still enabled, disable it. set global tidb_evolve_plan_baselines = off; +======= + SELECT @@global.tidb_evolve_plan_baselines; + + -- If `tidb_evolve_plan_baselines` is still enabled, disable it. + + SET GLOBAL tidb_evolve_plan_baselines = OFF; +>>>>>>> 4d59789e7 (update SPM doc for SQL plan auto capture (#7975)) ``` * Before you upgrade from v4.0 to the current version, you need to check whether the syntax of all queries corresponding to the available SQL bindings is correct in the new version. If any syntax errors exist, delete the corresponding SQL binding. To do that, perform the following steps. @@ -363,7 +518,11 @@ During cluster upgrade, SQL Plan Management (SPM) might cause compatibility issu ```sql -- Check the query corresponding to the available SQL binding in the version to be upgraded. +<<<<<<< HEAD select bind_sql from mysql.bind_info where status = 'using'; +======= + SELECT bind_sql FROM mysql.bind_info WHERE status = 'using'; +>>>>>>> 4d59789e7 (update SPM doc for SQL plan auto capture (#7975)) -- Verify the result from the above SQL query in the test environment of the new version. diff --git a/system-variables.md b/system-variables.md index c9c5ef0a651d8..c9ea759ba89ec 100644 --- a/system-variables.md +++ b/system-variables.md @@ -1228,6 +1228,15 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - This variable is used to set the memory usage threshold of the local cache in the `Apply` operator. - The local cache in the `Apply` operator is used to speed up the computation of the `Apply` operator. You can set the variable to `0` to disable the `Apply` cache feature. +### tidb_mem_quota_binding_cache (New in v6.0.0) + +- Scope: GLOBAL +- Default value: `67108864` (64 MiB) +- Range: `[0, 2147483647]` +- Unit: Bytes +- This variable is used to set the threshold of the memory used for caching bindings. +- If a system creates or captures excessive bindings, resulting in overuse of memory space, TiDB returns a warning in the log. In this case, the cache cannot hold all available bindings or determine which bindings to store. For this reason, some queries might miss their bindings. To address this problem, you can increase the value of this variable, which increases the memory used for caching bindings. After modifying this parameter, you need to run `admin reload bindings` to reload bindings and validate the modification. + ### tidb_mem_quota_query - Scope: SESSION From 14b14e4a725a4d78e8bc5910c1df57708f28c54d Mon Sep 17 00:00:00 2001 From: shichun-0415 Date: Sat, 2 Apr 2022 14:09:35 +0800 Subject: [PATCH 36/36] Revert "Merge branch 'master' into release-6.0" This reverts commit 9323f5a62d8f874665e5f6521823e3bb07f09e3e, reversing changes made to 0cadc1b706557160d16d1f315e52fd912ee83673. --- sql-plan-management.md | 315 +++++++++++------------------------------ system-variables.md | 9 -- 2 files changed, 84 insertions(+), 240 deletions(-) diff --git a/sql-plan-management.md b/sql-plan-management.md index 9b4f0bddd80ab..2c1d4fa039643 100644 --- a/sql-plan-management.md +++ b/sql-plan-management.md @@ -21,46 +21,42 @@ CREATE [GLOBAL | SESSION] BINDING FOR BindableStmt USING BindableStmt This statement binds SQL execution plans at the GLOBAL or SESSION level. Currently, supported bindable SQL statements (BindableStmt) in TiDB include `SELECT`, `DELETE`, `UPDATE`, and `INSERT` / `REPLACE` with `SELECT` subqueries. -> **Note:** -> -> Bindings have higher priority over manually added hints. Therefore, when you execute a statement containing a hint while a corresponding binding is present, the hint controlling the behavior of the optimizer does not take effect. However, other types of hints are still effective. - Specifically, two types of these statements cannot be bound to execution plans due to syntax conflicts. See the following examples: ```sql --- Type one: Statements that get the Cartesian product by using the `JOIN` keyword and not specifying the associated columns with the `USING` keyword. -CREATE GLOBAL BINDING for - SELECT * FROM t t1 JOIN t t2 -USING - SELECT * FROM t t1 JOIN t t2; - --- Type two: `DELETE` statements that contain the `USING` keyword. -CREATE GLOBAL BINDING for - DELETE FROM t1 USING t1 JOIN t2 ON t1.a = t2.a -USING - DELETE FROM t1 USING t1 JOIN t2 ON t1.a = t2.a; +-- Type one: Statements that get the Cartesian product by using the `join` keyword and not specifying the associated columns with the `using` keyword. +create global binding for + select * from t t1 join t t2 +using + select * from t t1 join t t2; + +-- Type two: `DELETE` statements that contain the `using` keyword. +create global binding for + delete from t1 using t1 join t2 on t1.a = t2.a +using + delete from t1 using t1 join t2 on t1.a = t2.a; ``` You can bypass syntax conflicts by using equivalent statements. For example, you can rewrite the above statements in the following ways: ```sql --- First rewrite of type one statements: Add a `USING` clause for the `JOIN` keyword. -CREATE GLOBAL BINDING for - SELECT * FROM t t1 JOIN t t2 USING (a) -USING - SELECT * FROM t t1 JOIN t t2 USING (a); - --- Second rewrite of type one statements: Delete the `JOIN` keyword. -CREATE GLOBAL BINDING for - SELECT * FROM t t1, t t2 -USING - SELECT * FROM t t1, t t2; - --- Rewrite of type two statements: Remove the `USING` keyword from the `delete` statement. -CREATE GLOBAL BINDING for - DELETE t1 FROM t1 JOIN t2 ON t1.a = t2.a +-- First rewrite of type one statements: Add a `using` clause for the `join` keyword. +create global binding for + select * from t t1 join t t2 using (a) using - DELETE t1 FROM t1 JOIN t2 ON t1.a = t2.a; + select * from t t1 join t t2 using (a); + +-- Second rewrite of type one statements: Delete the `join` keyword. +create global binding for + select * from t t1, t t2 +using + select * from t t1, t t2; + +-- Rewrite of type two statements: Remove the `using` keyword from the `delete` statement. +create global binding for + delete t1 from t1 join t2 on t1.a = t2.a +using + delete t1 from t1 join t2 on t1.a = t2.a; ``` > **Note:** @@ -71,16 +67,16 @@ Here are two examples: ```sql -- The hint takes effect in the following statement. -CREATE GLOBAL BINDING for - INSERT INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1 +create global binding for + insert into t1 select * from t2 where a > 1 and b = 1 using - INSERT INTO t1 SELECT /*+ use_index(@sel_1 t2, a) */ * FROM t2 WHERE a > 1 AND b = 1; + insert into t1 select /*+ use_index(@sel_1 t2, a) */ * from t2 where a > 1 and b = 1; -- The hint cannot take effect in the following statement. -CREATE GLOBAL BINDING for - INSERT INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1 +create global binding for + insert into t1 select * from t2 where a > 1 and b = 1 using - INSERT /*+ use_index(@sel_1 t2, a) */ INTO t1 SELECT * FROM t2 WHERE a > 1 AND b = 1; + insert /*+ use_index(@sel_1 t2, a) */ into t1 select * from t2 where a > 1 and b = 1; ``` If you do not specify the scope when creating an execution plan binding, the default scope is SESSION. The TiDB optimizer normalizes bound SQL statements and stores them in the system table. When processing SQL queries, if a normalized statement matches one of the bound SQL statements in the system table and the system variable `tidb_use_plan_baselines` is set to `on` (the default value is `on`), TiDB then uses the corresponding optimizer hint for this statement. If there are multiple matchable execution plans, the optimizer chooses the least costly one to bind. @@ -88,9 +84,9 @@ If you do not specify the scope when creating an execution plan binding, the def `Normalization` is a process that converts a constant in an SQL statement to a variable parameter and explicitly specifies the database for tables referenced in the query, with standardized processing on the spaces and line breaks in the SQL statement. See the following example: ```sql -SELECT * FROM t WHERE a > 1 +select * from t where a > 1 -- After normalization, the above statement is as follows: -SELECT * FROM test . t WHERE a > ? +select * from test . t where a > ? ``` > **Note:** @@ -100,15 +96,15 @@ SELECT * FROM test . t WHERE a > ? > For example: > > ```sql -> SELECT * FROM t limit 10 -> SELECT * FROM t limit 10, 20 -> SELECT * FROM t WHERE a IN (1) -> SELECT * FROM t WHERE a IN (1,2,3) +> select * from t limit 10 +> select * from t limit 10, 20 +> select * from t where a in (1) +> select * from t where a in (1,2,3) > -- After normalization, the above statements are as follows: -> SELECT * FROM test . t limit ? -> SELECT * FROM test . t limit ... -> SELECT * FROM test . t WHERE a IN ( ? ) -> SELECT * FROM test . t WHERE a IN ( ... ) +> select * from test . t limit ? +> select * from test . t limit ... +> select * from test . t where a in ( ? ) +> select * from test . t where a in ( ... ) > ``` > > When bindings are created, TiDB treats SQL statements that contain a single constant and SQL statements that contain multiple constants joined by commas differently. Therefore, you need to create bindings for the two SQL types separately. @@ -119,25 +115,25 @@ For example: ```sql -- Creates a GLOBAL binding and specifies using `sort merge join` in this binding. -CREATE GLOBAL BINDING for - SELECT * FROM t1, t2 WHERE t1.id = t2.id -USING - SELECT /*+ merge_join(t1, t2) */ * FROM t1, t2 WHERE t1.id = t2.id; +create global binding for + select * from t1, t2 where t1.id = t2.id +using + select /*+ merge_join(t1, t2) */ * from t1, t2 where t1.id = t2.id; -- The execution plan of this SQL statement uses the `sort merge join` specified in the GLOBAL binding. -explain SELECT * FROM t1, t2 WHERE t1.id = t2.id; +explain select * from t1, t2 where t1.id = t2.id; -- Creates another SESSION binding and specifies using `hash join` in this binding. -CREATE BINDING for - SELECT * FROM t1, t2 WHERE t1.id = t2.id -USING - SELECT /*+ hash_join(t1, t2) */ * FROM t1, t2 WHERE t1.id = t2.id; +create binding for + select * from t1, t2 where t1.id = t2.id +using + select /*+ hash_join(t1, t2) */ * from t1, t2 where t1.id = t2.id; -- In the execution plan of this statement, `hash join` specified in the SESSION binding is used, instead of `sort merge join` specified in the GLOBAL binding. -explain SELECT * FROM t1, t2 WHERE t1.id = t2.id; +explain select * from t1, t2 where t1.id = t2.id; ``` -When the first `SELECT` statement is being executed, the optimizer adds the `sm_join(t1, t2)` hint to the statement through the binding in the GLOBAL scope. The top node of the execution plan in the `explain` result is MergeJoin. When the second `SELECT` statement is being executed, the optimizer uses the binding in the SESSION scope instead of the binding in the GLOBAL scope and adds the `hash_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is HashJoin. +When the first `select` statement is being executed, the optimizer adds the `sm_join(t1, t2)` hint to the statement through the binding in the GLOBAL scope. The top node of the execution plan in the `explain` result is MergeJoin. When the second `select` statement is being executed, the optimizer uses the binding in the SESSION scope instead of the binding in the GLOBAL scope and adds the `hash_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is HashJoin. Each standardized SQL statement can have only one binding created using `CREATE BINDING` at a time. When multiple bindings are created for the same standardized SQL statement, the last created binding is retained, and all previous bindings (created and evolved) are marked as deleted. But session bindings and global bindings can coexist and are not affected by this logic. @@ -145,13 +141,13 @@ In addition, when you create a binding, TiDB requires that the session is in a d The original SQL statement and the bound statement must have the same text after normalization and hint removal, or the binding will fail. Take the following examples: -- This binding can be created successfully because the texts before and after parameterization and hint removal are the same: `SELECT * FROM test . t WHERE a > ?` +- This binding can be created successfully because the texts before and after parameterization and hint removal are the same: `select * from test . t where a > ?` ```sql CREATE BINDING FOR SELECT * FROM t WHERE a > 1 USING SELECT * FROM t use index (idx) WHERE a > 2 ``` -- This binding will fail because the original SQL statement is processed as `SELECT * FROM test . t WHERE a > ?`, while the bound SQL statement is processed differently as `SELECT * FROM test . t WHERE b > ?`. +- This binding will fail because the original SQL statement is processed as `select * from test . t where a > ?`, while the bound SQL statement is processed differently as `select * from test . t where b > ?`. ```sql CREATE BINDING FOR SELECT * FROM t WHERE a > 1 USING SELECT * FROM t use index(idx) WHERE b > 2 @@ -177,10 +173,10 @@ The following example is based on the example in [create binding](#create-a-bind ```sql -- Drops the binding created in the SESSION scope. -drop session binding for SELECT * FROM t1, t2 WHERE t1.id = t2.id; +drop session binding for select * from t1, t2 where t1.id = t2.id; -- Views the SQL execution plan again. -explain SELECT * FROM t1,t2 WHERE t1.id = t2.id; +explain select * from t1,t2 where t1.id = t2.id; ``` In the example above, the dropped binding in the SESSION scope shields the corresponding binding in the GLOBAL scope. The optimizer does not add the `sm_join(t1, t2)` hint to the statement. The top node of the execution plan in the `explain` result is not fixed to MergeJoin by this hint. Instead, the top node is independently selected by the optimizer according to the cost estimation. @@ -189,19 +185,7 @@ In the example above, the dropped binding in the SESSION scope shields the corre > > Executing `DROP GLOBAL BINDING` drops the binding in the current tidb-server instance cache and changes the status of the corresponding row in the system table to 'deleted'. This statement does not directly delete the records in the system table, because other tidb-server instances need to read the 'deleted' status to drop the corresponding binding in their cache. For the records in these system tables with the status of 'deleted', at every 100 `bind-info-lease` (the default value is `3s`, and `300s` in total) interval, the background thread triggers an operation of reclaiming and clearing on the bindings of `update_time` before 10 `bind-info-lease` (to ensure that all tidb-server instances have read the 'deleted' status and updated the cache). -## Change binding status - -{{< copyable "sql" >}} - -```sql -SET BINDING [ENABLED | DISABLED] FOR BindableStmt; -``` - -You can execute this statement to change the status of a binding. The default status is ENABLED. The effective scope is GLOBAL by default and cannot be modified. - -When executing this statement, you can only change the status of a binding from `Disabled` to `Enabled` or from `Enabled` to `Disabled`. If no binding is available for status changes, a warning message is returned, saying `There are no bindings can be set the status. Please check the SQL text`. Note that a binding in `Disabled` status is not used by any query. - -### View bindings +### View binding {{< copyable "sql" >}} @@ -216,98 +200,41 @@ This statement outputs the execution plan bindings at the GLOBAL or SESSION leve | original_sql | Original SQL statement after parameterization | | bind_sql | Bound SQL statement with hints | | default_db | Default database | -| status | Status including Enabled (replacing the Using status from v6.0), Disabled, Deleted, Invalid, Rejected, and Pending verification| +| status | Status including Using, Deleted, Invalid, Rejected, and Pending verification| | create_time | Creating time | | update_time | Updating time | | charset | Character set | | collation | Ordering rule | | source | The way in which a binding is created, including `manual` (created by the `create [global] binding` SQL statement), `capture` (captured automatically by TiDB), and `evolve` (evolved automatically by TiDB) | -### Troubleshoot a binding - -You can use either of the following methods to troubleshoot a binding: - -- Use the system variable [`last_plan_from_binding`](/system-variables.md#last_plan_from_binding-new-in-v40) to show whether the execution plan used by the last executed statement is from the binding. - - {{< copyable "sql" >}} - - ```sql - -- Create a global binding - CREATE GLOBAL BINDING for - SELECT * FROM t - USING - SELECT /*+ USE_INDEX(t, idx_a) */ * FROM t; - - SELECT * FROM t; - SELECT @@[SESSION.]last_plan_from_binding; - ``` - - ```sql - +--------------------------+ - | @@last_plan_from_binding | - +--------------------------+ - | 1 | - +--------------------------+ - 1 row in set (0.00 sec) - ``` - -- Use the `explain format = 'verbose'` statement to view the query plan of a SQL statement. If the SQL statement uses a binding, you can run `show warnings` to check which binding is used in the SQL statement. - - ```sql - -- Create a global binding - - CREATE GLOBAL BINDING for - SELECT * FROM t - USING - SELECT /*+ USE_INDEX(t, idx_a) */ * FROM t; - - -- Use explain format = 'verbose' to view the execution plan of a SQL statement - - explain format = 'verbose' SELECT * FROM t; - - -- Run `show warnings` to view the binding used in the query. - - show warnings; - ``` - - ```sql - +-------+------+--------------------------------------------------------------------------+ - | Level | Code | Message | - +-------+------+--------------------------------------------------------------------------+ - | Note | 1105 | Using the bindSQL: SELECT /*+ USE_INDEX(`t` `idx_a`)*/ * FROM `test`.`t` | - +-------+------+--------------------------------------------------------------------------+ - 1 row in set (0.01 sec) - - ``` +### Troubleshoot binding -### Cache bindings +{{< copyable "sql" >}} -Each TiDB instance has a least recently used (LRU) cache for bindings. The cache capacity is controlled by the system variable [`tidb_mem_quota_binding_cache`](/system-variables.md#tidb_mem_quota_binding_cache-new-in-v600). You can view bindings that are cached in the TiDB instance. +```sql +SELECT @@[SESSION.]last_plan_from_binding; +``` -To view the cache status of bindings, run the `SHOW binding_cache status` statement. In this statement, the effective scope is GLOBAL by default and cannot be modified. This statement returns the number of available bindings in the cache, the total number of available bindings in the system, memory usage of all cached bindings, and the total memory for the cache. +This statement uses the system variable [`last_plan_from_binding`](/system-variables.md#last_plan_from_binding-new-in-v40) to show whether the execution plan used by the last executed statement is from the binding. -{{< copyable "sql" >}} +In addition, when you use the `explain format = 'verbose'` statement to view the query plan of a SQL statement, if the SQL statement uses binding, the `explain` statement will return a warning. In this situation, you can check the warning message to learn which binding is used in the SQL statement. ```sql +-- Create a global binding. -SHOW binding_cache status; -``` +create global binding for + select * from t +using + select * from t; -```sql -+-------------------+-------------------+--------------+--------------+ -| bindings_in_cache | bindings_in_table | memory_usage | memory_quota | -+-------------------+-------------------+--------------+--------------+ -| 1 | 1 | 159 Bytes | 64 MB | -+-------------------+-------------------+--------------+--------------+ -1 row in set (0.00 sec) +-- Use the `explain format = 'verbose'` statement to check the SQL execution plan. Check the warning message to view the binding used in the query. + +explain format = 'verbose' select * from t; +show warnings; ``` ## Baseline capturing -Used for [preventing regression of execution plans during an upgrade](#prevent-regression-of-execution-plans-during-an-upgrade), this feature captures queries that meet capturing conditions and creates bindings for these queries. - -### Enable capturing - To enable baseline capturing, set `tidb_capture_plan_baselines` to `on`. The default value is `off`. > **Note:** @@ -320,12 +247,7 @@ However, TiDB does not automatically capture bindings for the following types of - `EXPLAIN` and `EXPLAIN ANALYZE` statements. - SQL statements executed internally in TiDB, such as `SELECT` queries used for automatically loading statistical information. -- Statements that contain `Enabled` or `Disabled` bindings. -- Statements that are filtered out by capturing conditions. - -> **Note:** -> -> Currently, a binding generates a group of hints to fix an execution plan generated by a query statement. In this way, for the same query, the execution plan does not change. For most OLTP queries, including queries using the same index or Join algorithm (such as HashJoin and IndexJoin), TiDB guarantees plan consistency before and after the binding. However, due to the limitations of hints, TiDB cannot guarantee plan consistency for some complex queries, such as Join of more than two tables, MPP queries, and complex OLAP queries. +- SQL statements that are bound to a manually created execution plan. For `PREPARE` / `EXECUTE` statements and for queries executed with binary protocols, TiDB automatically captures bindings for the real query statements, not for the `PREPARE` / `EXECUTE` statements. @@ -333,75 +255,6 @@ For `PREPARE` / `EXECUTE` statements and for queries executed with binary protoc > > Because TiDB has some embedded SQL statements to ensure the correctness of some features, baseline capturing by default automatically shields these SQL statements. -### Filter out bindings - -This feature allows you to configure a blocklist to filter out queries whose bindings you do not want to capture. A blocklist has three dimensions, table name, frequency, and user name. - -#### Usage - -Insert filtering conditions into the system table `mysql.capture_plan_baselines_blacklist`. Then the filtering conditions take effect in the entire cluster immediately. - -```sql --- Filter by table name - INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'test.t'); - --- Filter by database name and table name through wildcards - INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'test.table_*'); - INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('table', 'db_*.table_*'); - --- Filter by frequency - INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('frequency', '2'); - --- Filter by user name - INSERT INTO mysql.capture_plan_baselines_blacklist(filter_type, filter_value) VALUES('user', 'user1'); -``` - -| **Dimension name** | **Description** | Remarks | -| :----------- | :----------------------------------------------------------- | ------------------------------------------------------------ | -| table | Filter by table name. Each filtering rule is in the `db.table` format. The supported filtering syntax includes [Plain table names](/table-filter.md#plain-table-names) and [Wildcards](/table-filter.md#wildcards). | Case insensitive. If a table name contains illegal characters, the log returns a warning message `[sql-bind] failed to load mysql.capture_plan_baselines_blacklist`. | -| frequency | Filter by frequency. SQL statements executed more than once are captured by default. You can set a high frequency to capture statements that are frequently executed. | Setting frequency to a value smaller than 1 is considered illegal, and the log returns a warning message `[sql-bind] frequency threshold is less than 1, ignore it`. If multiple frequency filter rules are inserted, the value with the highest frequency prevails. | -| user | Filter by user name. Statements executed by blocklisted users are not captured. | If multiple users execute the same statement and their user names are all in the blocklist, this statement is not captured. | - -> **Note:** -> -> - Modifying a blocklist requires the super privilege. -> -> - If a blocklist contains illegal filters, TiDB returns the warning message `[sql-bind] unknown capture filter type, ignore it` in the log. - -### Prevent regression of execution plans during an upgrade - - Before upgrading a TiDB cluster, you can use baseline capturing to prevent regression of execution plans by performing the following steps: - -1. Enable baseline capturing and keep it working for a period of time. - - > **Note:** - > - > Test data shows that long-term working of baseline capturing has a slight impact on the performance of the cluster load. Therefore, it is recommended to enable baseline capturing as long as possible so that important plans (appear twice or above) are captured. - -2. Upgrade the TiDB cluster. After the upgrade, TiDB uses those captured bindings to ensure execution plan consistency. - -3. After the upgrade, delete bindings as required. - - - Check the binding source by running the [`SHOW GLOBAL BINDINGS`](#view-bindings) statement. - - In the output, check the `Source` field to see whether a binding is captured (`capture`) or manually created (`manual`). - - - Determine whether to retain the captured bindings: - - ``` - -- View the plan with the binding enabled - SET @@SESSION.TIDB_USE_PLAN_BASELINES = true; - EXPLAIN FORMAT='VERBOSE' SELECT * FROM t1 WHERE ...; - - -- View the plan with the binding disabled - SET @@SESSION.TIDB_USE_PLAN_BASELINES = false; - EXPLAIN FORMAT='VERBOSE' SELECT * FROM t1 WHERE ...; - ``` - - - If the execution plan is consistent, you can delete the binding safely. - - - If the execution plan is inconsistent, you need to identify the cause, for example, by checking statistics. In this case, you need to retain the binding to ensure plan consistency. - ## Baseline evolution Baseline evolution is an important feature of SPM introduced in TiDB v4.0. @@ -417,7 +270,7 @@ Use the following statement to enable automatic binding evolution: {{< copyable "sql" >}} ```sql -SET GLOBAL tidb_evolve_plan_baselines = ON; +set global tidb_evolve_plan_baselines = on; ``` The default value of `tidb_evolve_plan_baselines` is `off`. @@ -434,7 +287,7 @@ Assume that table `t` is defined as follows: {{< copyable "sql" >}} ```sql -CREATE TABLE t(a INT, b INT, KEY(a), KEY(b)); +create table t(a int, b int, key(a), key(b)); ``` Perform the following query on table `t`: @@ -442,7 +295,7 @@ Perform the following query on table `t`: {{< copyable "sql" >}} ```sql -SELECT * FROM t WHERE a < 100 AND b < 100; +select * from t where a < 100 and b < 100; ``` In the table defined above, few rows meet the `a < 100` condition. But for some reason, the optimizer mistakenly selects the full table scan instead of the optimal execution plan that uses index `a`. You can first use the following statement to create a binding: @@ -450,7 +303,7 @@ In the table defined above, few rows meet the `a < 100` condition. But for some {{< copyable "sql" >}} ```sql -CREATE GLOBAL BINDING for SELECT * FROM t WHERE a < 100 AND b < 100 using SELECT * FROM t use index(a) WHERE a < 100 AND b < 100; +create global binding for select * from t where a < 100 and b < 100 using select * from t use index(a) where a < 100 and b < 100; ``` When the query above is executed again, the optimizer selects index `a` (influenced by the binding created above) to reduce the query time. @@ -496,11 +349,11 @@ During cluster upgrade, SQL Plan Management (SPM) might cause compatibility issu ```sql -- Check whether `tidb_evolve_plan_baselines` is disabled in the earlier version. - SELECT @@global.tidb_evolve_plan_baselines; + select @@global.tidb_evolve_plan_baselines; -- If `tidb_evolve_plan_baselines` is still enabled, disable it. - SET GLOBAL tidb_evolve_plan_baselines = OFF; + set global tidb_evolve_plan_baselines = off; ``` * Before you upgrade from v4.0 to the current version, you need to check whether the syntax of all queries corresponding to the available SQL bindings is correct in the new version. If any syntax errors exist, delete the corresponding SQL binding. To do that, perform the following steps. @@ -510,7 +363,7 @@ During cluster upgrade, SQL Plan Management (SPM) might cause compatibility issu ```sql -- Check the query corresponding to the available SQL binding in the version to be upgraded. - SELECT bind_sql FROM mysql.bind_info WHERE status = 'using'; + select bind_sql from mysql.bind_info where status = 'using'; -- Verify the result from the above SQL query in the test environment of the new version. diff --git a/system-variables.md b/system-variables.md index c9ea759ba89ec..c9c5ef0a651d8 100644 --- a/system-variables.md +++ b/system-variables.md @@ -1228,15 +1228,6 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - This variable is used to set the memory usage threshold of the local cache in the `Apply` operator. - The local cache in the `Apply` operator is used to speed up the computation of the `Apply` operator. You can set the variable to `0` to disable the `Apply` cache feature. -### tidb_mem_quota_binding_cache (New in v6.0.0) - -- Scope: GLOBAL -- Default value: `67108864` (64 MiB) -- Range: `[0, 2147483647]` -- Unit: Bytes -- This variable is used to set the threshold of the memory used for caching bindings. -- If a system creates or captures excessive bindings, resulting in overuse of memory space, TiDB returns a warning in the log. In this case, the cache cannot hold all available bindings or determine which bindings to store. For this reason, some queries might miss their bindings. To address this problem, you can increase the value of this variable, which increases the memory used for caching bindings. After modifying this parameter, you need to run `admin reload bindings` to reload bindings and validate the modification. - ### tidb_mem_quota_query - Scope: SESSION