From 633be89feb39b682c0db60d41011527f21e668d9 Mon Sep 17 00:00:00 2001 From: Liuxiaozhen12 Date: Fri, 18 Jun 2021 15:58:19 +0800 Subject: [PATCH 1/4] update docs related to partition table dynamic mode --- experimental-features.md | 1 + partitioned-table.md | 192 +++++++++++++++++++++++++++++++++++++++ system-variables.md | 10 ++ 3 files changed, 203 insertions(+) diff --git a/experimental-features.md b/experimental-features.md index 94594876e93be..d76a87c38b083 100644 --- a/experimental-features.md +++ b/experimental-features.md @@ -24,6 +24,7 @@ This document introduces the experimental features of TiDB in different versions + List Partition (Introduced in v5.0) + List COLUMNS Partition (Introduced in v5.0) ++ [Partition Table Dynamic Mode](/partitioned-table.md#dynamic-mode). (Introduced in v5.1) + The expression index feature. The expression index is also called the function-based index. When you create an index, the index fields do not have to be a specific column but can be an expression calculated from one or more columns. This feature is useful for quickly accessing the calculation-based tables. See [Expression index](/sql-statements/sql-statement-create-index.md) for details. (Introduced in v4.0) + [Generated Columns](/generated-columns.md). + [User-Defined Variables](/user-defined-variables.md). diff --git a/partitioned-table.md b/partitioned-table.md index d06b0977a08a4..e2ad58862e9e1 100644 --- a/partitioned-table.md +++ b/partitioned-table.md @@ -1240,3 +1240,195 @@ select * from t; The `tidb_enable_list_partition` environment variable controls whether to enable the partitioned table feature. If this variable is set to `OFF`, the partition information will be ignored when a table is created, and this table will be created as a normal table. This variable is only used in table creation. After the table is created, modify this variable value takes no effect. For details, see [system variables](/system-variables.md#tidb_enable_list_partition-new-in-v50). + +### Dynamic mode + +> **Warning:** +> +> This is still an experimental feature. It is **NOT** recommended that you use it in the production environment. + +There are two modes for TiDB accessing partitioned tables: `dynamic` and `static`. Currently, the `static` mode is used by default. If you want to enable `dynamic` mode, you need to manually set `tidb_partition_prune_mode` to `dynamic`. + +{{< copyable "sql" >}} + +```sql +set @@session.tidb_partition_prune_mode = 'dynamic' +``` + +In `static` mode, TiDB accesses each partition separately with multiple operators, and then merges the results through `Union`. The following example is a simple read operation, where you can find that TiDB merges the results of two corresponding partitions with `Union`: + +{{< copyable "sql" >}} + +```sql +mysql> create table t1(id int, age int, key(id)) partition by range(id) ( + -> partition p0 values less than (100), + -> partition p1 values less than (200), + -> partition p2 values less than (300), + -> partition p3 values less than (400)); +Query OK, 0 rows affected (0.01 sec) + +mysql> explain select * from t1 where id < 150; ++------------------------------+----------+-----------+------------------------+--------------------------------+ +| id | estRows | task | access object | operator info | ++------------------------------+----------+-----------+------------------------+--------------------------------+ +| PartitionUnion_9 | 6646.67 | root | | | +| ├─TableReader_12 | 3323.33 | root | | data:Selection_11 | +| │ └─Selection_11 | 3323.33 | cop[tikv] | | lt(test.t1.id, 150) | +| │ └─TableFullScan_10 | 10000.00 | cop[tikv] | table:t1, partition:p0 | keep order:false, stats:pseudo | +| └─TableReader_18 | 3323.33 | root | | data:Selection_17 | +| └─Selection_17 | 3323.33 | cop[tikv] | | lt(test.t1.id, 150) | +| └─TableFullScan_16 | 10000.00 | cop[tikv] | table:t1, partition:p1 | keep order:false, stats:pseudo | ++------------------------------+----------+-----------+------------------------+--------------------------------+ +7 rows in set (0.00 sec) +``` + +In `dynamic` mode, each operator supports direct access to multiple partitions, so TiDB no longer uses `Union`. + +{{< copyable "sql" >}} + +```sql +mysql> set @@session.tidb_partition_prune_mode = 'dynamic'; +Query OK, 0 rows affected (0.00 sec) + +mysql> explain select * from t1 where id < 150; ++-------------------------+----------+-----------+-----------------+--------------------------------+ +| id | estRows | task | access object | operator info | ++-------------------------+----------+-----------+-----------------+--------------------------------+ +| TableReader_7 | 3323.33 | root | partition:p0,p1 | data:Selection_6 | +| └─Selection_6 | 3323.33 | cop[tikv] | | lt(test.t1.id, 150) | +| └─TableFullScan_5 | 10000.00 | cop[tikv] | table:t1 | keep order:false, stats:pseudo | ++-------------------------+----------+-----------+-----------------+--------------------------------+ +3 rows in set (0.00 sec) +``` + +From the above query results, it can be seen that the `Union` opearator in the execution plan disappears while the partition pruning is still in effect and the execution plan only accesses `p0` and `p1`. + +`dynamic` mode makes execution plans more simple and clear. Omitting the Union operation can improve the execution efficiency and avoid the problem of Union concurrent execution. In addition, `dynamic` mode also solves two problems that cannot be solved in `static` mode: + ++ Do not use Plan Cache (see example 1 and 2) ++ Do not use IndexJoin type (see example 3 and 4) + +**Example 1**：The following example enables Plan Cache function in the configuration file and executes the same query twice in `static` mode: + +{{< copyable "sql" >}} + +```sql +mysql> set @a=150; +Query OK, 0 rows affected (0.00 sec) + +mysql> set @@tidb_partition_prune_mode = 'static'; +Query OK, 0 rows affected (0.00 sec) + +mysql> prepare stmt from 'select * from t1 where id < ?'; +Query OK, 0 rows affected (0.00 sec) + +mysql> execute stmt using @a; +Empty set (0.00 sec) + +mysql> execute stmt using @a; +Empty set (0.00 sec) + +-- In static mode, when the same query is executed twice, the cache cannot be hit the second time +mysql> select @@last_plan_from_cache; ++------------------------+ +| @@last_plan_from_cache | ++------------------------+ +| 0 | ++------------------------+ +1 row in set (0.00 sec) +``` + +`last_plan_from_cache` variable can display whether the last query hit the Plan Cache. As can be seen from above, in `static` mode, even if the same query is executed multiple times on the partitioned table, the Plan Cache will not be hit. + +**Example 2**: The following example performs the same operations as example 1 in `dynamic` mode: + +{{< copyable "sql" >}} + +```sql +mysql> set @@tidb_partition_prune_mode = 'dynamic'; +Query OK, 0 rows affected (0.00 sec) + +mysql> prepare stmt from 'select * from t1 where id < ?'; +Query OK, 0 rows affected (0.00 sec) + +mysql> execute stmt using @a; +Empty set (0.00 sec) + +mysql> execute stmt using @a; +Empty set (0.00 sec) + +-- in dynamic mode, the cache can be hit the second time +mysql> select @@last_plan_from_cache; ++------------------------+ +| @@last_plan_from_cache | ++------------------------+ +| 1 | ++------------------------+ +1 row in set (0.00 sec) +``` + +As can be seen from above, in `dynamic` mode, querying partitioned tables can hit the Plan Cache. + +**Example 3**: The following example tries the execution plan with IndexJoin query in `static` mode: + +{{< copyable "sql" >}} + +```sql +mysql> create table t2(id int, code int); +Query OK, 0 rows affected (0.01 sec) + +mysql> set @@tidb_partition_prune_mode = 'static'; +Query OK, 0 rows affected (0.00 sec) + +mysql> explain select /*+ TIDB_INLJ(t1, t2) */ t1.* from t1, t2 where t2.code = 0 and t2.id = t1.id; ++--------------------------------+----------+-----------+------------------------+------------------------------------------------+ +| id | estRows | task | access object | operator info | ++--------------------------------+----------+-----------+------------------------+------------------------------------------------+ +| HashJoin_13 | 12.49 | root | | inner join, equal:[eq(test.t1.id, test.t2.id)] | +| ├─TableReader_42(Build) | 9.99 | root | | data:Selection_41 | +| │ └─Selection_41 | 9.99 | cop[tikv] | | eq(test.t2.code, 0), not(isnull(test.t2.id)) | +| │ └─TableFullScan_40 | 10000.00 | cop[tikv] | table:t2 | keep order:false, stats:pseudo | +| └─PartitionUnion_15(Probe) | 39960.00 | root | | | +| ├─TableReader_18 | 9990.00 | root | | data:Selection_17 | +| │ └─Selection_17 | 9990.00 | cop[tikv] | | not(isnull(test.t1.id)) | +| │ └─TableFullScan_16 | 10000.00 | cop[tikv] | table:t1, partition:p0 | keep order:false, stats:pseudo | +| ├─TableReader_24 | 9990.00 | root | | data:Selection_23 | +| │ └─Selection_23 | 9990.00 | cop[tikv] | | not(isnull(test.t1.id)) | +| │ └─TableFullScan_22 | 10000.00 | cop[tikv] | table:t1, partition:p1 | keep order:false, stats:pseudo | +| ├─TableReader_30 | 9990.00 | root | | data:Selection_29 | +| │ └─Selection_29 | 9990.00 | cop[tikv] | | not(isnull(test.t1.id)) | +| │ └─TableFullScan_28 | 10000.00 | cop[tikv] | table:t1, partition:p2 | keep order:false, stats:pseudo | +| └─TableReader_36 | 9990.00 | root | | data:Selection_35 | +| └─Selection_35 | 9990.00 | cop[tikv] | | not(isnull(test.t1.id)) | +| └─TableFullScan_34 | 10000.00 | cop[tikv] | table:t1, partition:p3 | keep order:false, stats:pseudo | ++--------------------------------+----------+-----------+------------------------+------------------------------------------------+ +17 rows in set, 1 warning (0.00 sec) +``` + +As can be seen from above, even if the hint of `TIDB_INLJ` is used, the query with partitioned table can not select the execution plan with IndexJoin. + +**Example 4**: The following example tries the execution plan with IndexJoin query in `dynamic` mode: + +{{< copyable "sql" >}} + +```sql +mysql> set @@tidb_partition_prune_mode = 'dynamic'; +Query OK, 0 rows affected (0.00 sec) + +mysql> explain select /*+ TIDB_INLJ(t1, t2) */ t1.* from t1, t2 where t2.code = 0 and t2.id = t1.id; ++---------------------------------+----------+-----------+------------------------+---------------------------------------------------------------------------------------------------------------------+ +| id | estRows | task | access object | operator info | ++---------------------------------+----------+-----------+------------------------+---------------------------------------------------------------------------------------------------------------------+ +| IndexJoin_11 | 12.49 | root | | inner join, inner:IndexLookUp_10, outer key:test.t2.id, inner key:test.t1.id, equal cond:eq(test.t2.id, test.t1.id) | +| ├─TableReader_16(Build) | 9.99 | root | | data:Selection_15 | +| │ └─Selection_15 | 9.99 | cop[tikv] | | eq(test.t2.code, 0), not(isnull(test.t2.id)) | +| │ └─TableFullScan_14 | 10000.00 | cop[tikv] | table:t2 | keep order:false, stats:pseudo | +| └─IndexLookUp_10(Probe) | 1.25 | root | partition:all | | +| ├─Selection_9(Build) | 1.25 | cop[tikv] | | not(isnull(test.t1.id)) | +| │ └─IndexRangeScan_7 | 1.25 | cop[tikv] | table:t1, index:id(id) | range: decided by [eq(test.t1.id, test.t2.id)], keep order:false, stats:pseudo | +| └─TableRowIDScan_8(Probe) | 1.25 | cop[tikv] | table:t1 | keep order:false, stats:pseudo | ++---------------------------------+----------+-----------+------------------------+---------------------------------------------------------------------------------------------------------------------+ +8 rows in set (0.00 sec) +``` + +As can be seen from above, in `dynamic` mode, the plan with IndexJoin is selected when you execute query. \ No newline at end of file diff --git a/system-variables.md b/system-variables.md index d58b285ef9c5d..2a97cf9539036 100644 --- a/system-variables.md +++ b/system-variables.md @@ -501,6 +501,16 @@ Constraint checking is always performed in place for pessimistic transactions (d - Default value: `OFF` - This variable is used to set whether to enable the `LIST (COLUMNS) TABLE PARTITION` feature. +### `tidb_partition_prune_mode` New in v5.1 + +> **Warning:** +> +> Currently, the dynamic mode for partitioned tables is an experimental feature. It is not recommended that you use it in the production environment. + +- Scope: SESSION | GLOBAL +- Default value: `static` +- This variable is used to set whether to enable `dynamic` mode for partitioned tables. For more details about dynamic mode, see [Partition Table Dynamic Mode](/partitioned-table.md#dynamic-mode). + ### tidb_enable_noop_functions New in v4.0 - Scope: SESSION | GLOBAL From cdc409fdcb8297661577215d9b0d9f574c2baff1 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 21 Jun 2021 10:35:31 +0800 Subject: [PATCH 2/4] Apply suggestions from code review --- experimental-features.md | 2 +- partitioned-table.md | 30 +++++++++++++++--------------- system-variables.md | 2 +- 3 files changed, 17 insertions(+), 17 deletions(-) diff --git a/experimental-features.md b/experimental-features.md index d76a87c38b083..9f77f5cf18f9f 100644 --- a/experimental-features.md +++ b/experimental-features.md @@ -24,7 +24,7 @@ This document introduces the experimental features of TiDB in different versions + List Partition (Introduced in v5.0) + List COLUMNS Partition (Introduced in v5.0) -+ [Partition Table Dynamic Mode](/partitioned-table.md#dynamic-mode). (Introduced in v5.1) ++ [Dynamic Mode for Partitioned Tables](/partitioned-table.md#dynamic-mode). (Introduced in v5.1) + The expression index feature. The expression index is also called the function-based index. When you create an index, the index fields do not have to be a specific column but can be an expression calculated from one or more columns. This feature is useful for quickly accessing the calculation-based tables. See [Expression index](/sql-statements/sql-statement-create-index.md) for details. (Introduced in v4.0) + [Generated Columns](/generated-columns.md). + [User-Defined Variables](/user-defined-variables.md). diff --git a/partitioned-table.md b/partitioned-table.md index e2ad58862e9e1..63a9f8d5b21e9 100644 --- a/partitioned-table.md +++ b/partitioned-table.md @@ -1247,7 +1247,7 @@ This variable is only used in table creation. After the table is created, modify > > This is still an experimental feature. It is **NOT** recommended that you use it in the production environment. -There are two modes for TiDB accessing partitioned tables: `dynamic` and `static`. Currently, the `static` mode is used by default. If you want to enable `dynamic` mode, you need to manually set `tidb_partition_prune_mode` to `dynamic`. +TiDB accesses partitioned tables in one of the two modes: `dynamic` mode and `static` mode. Currently, `static` mode is used by default. If you want to enable `dynamic` mode, you need to manually set the `tidb_partition_prune_mode` variable to `dynamic`. {{< copyable "sql" >}} @@ -1255,7 +1255,7 @@ There are two modes for TiDB accessing partitioned tables: `dynamic` and `static set @@session.tidb_partition_prune_mode = 'dynamic' ``` -In `static` mode, TiDB accesses each partition separately with multiple operators, and then merges the results through `Union`. The following example is a simple read operation, where you can find that TiDB merges the results of two corresponding partitions with `Union`: +In `static` mode, TiDB accesses each partition separately using multiple operators, and then merges the results using `Union`. The following example is a simple read operation where TiDB merges the results of two corresponding partitions using `Union`: {{< copyable "sql" >}} @@ -1301,14 +1301,14 @@ mysql> explain select * from t1 where id < 150; 3 rows in set (0.00 sec) ``` -From the above query results, it can be seen that the `Union` opearator in the execution plan disappears while the partition pruning is still in effect and the execution plan only accesses `p0` and `p1`. +From the above query results, you can see that the `Union` operator in the execution plan disappears while the partition pruning still takes effect and the execution plan only accesses `p0` and `p1`. -`dynamic` mode makes execution plans more simple and clear. Omitting the Union operation can improve the execution efficiency and avoid the problem of Union concurrent execution. In addition, `dynamic` mode also solves two problems that cannot be solved in `static` mode: +`dynamic` mode makes execution plans simpler and clearer. Omitting the Union operation can improve the execution efficiency and avoid the problem of Union concurrent execution. In addition, `dynamic` mode also solves two problems that cannot be solved in `static` mode: -+ Do not use Plan Cache (see example 1 and 2) -+ Do not use IndexJoin type (see example 3 and 4) ++ Plan Cache cannot be used. (See example 1 and 2) ++ Execution plans with IndexJoin cannot be used. (See example 3 and 4) -**Example 1**：The following example enables Plan Cache function in the configuration file and executes the same query twice in `static` mode: +**Example 1**：In the following example, the Plan Cache feature is enabled in the configuration file and the same query is executed twice in `static` mode: {{< copyable "sql" >}} @@ -1328,7 +1328,7 @@ Empty set (0.00 sec) mysql> execute stmt using @a; Empty set (0.00 sec) --- In static mode, when the same query is executed twice, the cache cannot be hit the second time +-- In static mode, when the same query is executed twice, the cache cannot be hit at the second time. mysql> select @@last_plan_from_cache; +------------------------+ | @@last_plan_from_cache | @@ -1338,9 +1338,9 @@ mysql> select @@last_plan_from_cache; 1 row in set (0.00 sec) ``` -`last_plan_from_cache` variable can display whether the last query hit the Plan Cache. As can be seen from above, in `static` mode, even if the same query is executed multiple times on the partitioned table, the Plan Cache will not be hit. +The `last_plan_from_cache` variable can show whether the last query hits the Plan Cache or not. From example 1, you can see that in `static` mode, even if the same query is executed multiple times on the partitioned table, the Plan Cache is not be hit. -**Example 2**: The following example performs the same operations as example 1 in `dynamic` mode: +**Example 2**: In the following example, the same operations are performed in `dynamic` mode as done in example 1: {{< copyable "sql" >}} @@ -1357,7 +1357,7 @@ Empty set (0.00 sec) mysql> execute stmt using @a; Empty set (0.00 sec) --- in dynamic mode, the cache can be hit the second time +-- In dynamic mode, the cache can be hit at the second time. mysql> select @@last_plan_from_cache; +------------------------+ | @@last_plan_from_cache | @@ -1367,7 +1367,7 @@ mysql> select @@last_plan_from_cache; 1 row in set (0.00 sec) ``` -As can be seen from above, in `dynamic` mode, querying partitioned tables can hit the Plan Cache. +From example 2, you can see that in `dynamic` mode, querying the partitioned table hits the Plan Cache. **Example 3**: The following example tries the execution plan with IndexJoin query in `static` mode: @@ -1405,9 +1405,9 @@ mysql> explain select /*+ TIDB_INLJ(t1, t2) */ t1.* from t1, t2 where t2.code = 17 rows in set, 1 warning (0.00 sec) ``` -As can be seen from above, even if the hint of `TIDB_INLJ` is used, the query with partitioned table can not select the execution plan with IndexJoin. +From example 3, you can see that even if the `TIDB_INLJ` hint is used, the query on the partitioned table cannot select the execution plan with IndexJoin. -**Example 4**: The following example tries the execution plan with IndexJoin query in `dynamic` mode: +**Example 4**: In the following example, the query is performed in `dynamic` mode using the execution plan with IndexJoin: {{< copyable "sql" >}} @@ -1431,4 +1431,4 @@ mysql> explain select /*+ TIDB_INLJ(t1, t2) */ t1.* from t1, t2 where t2.code = 8 rows in set (0.00 sec) ``` -As can be seen from above, in `dynamic` mode, the plan with IndexJoin is selected when you execute query. \ No newline at end of file +From example 4, you can see that in `dynamic` mode, the execution plan with IndexJoin is selected when you execute the query. diff --git a/system-variables.md b/system-variables.md index 2a97cf9539036..451013d360ecd 100644 --- a/system-variables.md +++ b/system-variables.md @@ -509,7 +509,7 @@ Constraint checking is always performed in place for pessimistic transactions (d - Scope: SESSION | GLOBAL - Default value: `static` -- This variable is used to set whether to enable `dynamic` mode for partitioned tables. For more details about dynamic mode, see [Partition Table Dynamic Mode](/partitioned-table.md#dynamic-mode). +- Specifies whether to enable `dynamic` mode for partitioned tables. For details about the dynamic mode, see [Dynamic Mode for Partitioned Tables](/partitioned-table.md#dynamic-mode). ### tidb_enable_noop_functions New in v4.0 From bed77a01e2152e3d608819b91eb9589a016f89cc Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 21 Jun 2021 10:35:40 +0800 Subject: [PATCH 3/4] Update partitioned-table.md --- partitioned-table.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/partitioned-table.md b/partitioned-table.md index 63a9f8d5b21e9..e638c9ceeead5 100644 --- a/partitioned-table.md +++ b/partitioned-table.md @@ -1369,7 +1369,7 @@ mysql> select @@last_plan_from_cache; From example 2, you can see that in `dynamic` mode, querying the partitioned table hits the Plan Cache. -**Example 3**: The following example tries the execution plan with IndexJoin query in `static` mode: +**Example 3**: In the following example, a query is performed in `static` mode using the execution plan with IndexJoin: {{< copyable "sql" >}} From c78077d76930d914536ef302a8bc7b020ae71293 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 21 Jun 2021 10:59:29 +0800 Subject: [PATCH 4/4] Update partitioned-table.md --- partitioned-table.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/partitioned-table.md b/partitioned-table.md index e638c9ceeead5..c6b8a198b675c 100644 --- a/partitioned-table.md +++ b/partitioned-table.md @@ -1338,7 +1338,7 @@ mysql> select @@last_plan_from_cache; 1 row in set (0.00 sec) ``` -The `last_plan_from_cache` variable can show whether the last query hits the Plan Cache or not. From example 1, you can see that in `static` mode, even if the same query is executed multiple times on the partitioned table, the Plan Cache is not be hit. +The `last_plan_from_cache` variable can show whether the last query hits the Plan Cache or not. From example 1, you can see that in `static` mode, even if the same query is executed multiple times on the partitioned table, the Plan Cache is not hit. **Example 2**: In the following example, the same operations are performed in `dynamic` mode as done in example 1: