From 3ff4a7032decdc416da5af322e159342b772ba43 Mon Sep 17 00:00:00 2001 From: en-jin19 Date: Tue, 17 Aug 2021 19:06:27 +0200 Subject: [PATCH 01/10] SQL statement, TiDB configuration: update expression indexes --- sql-statements/sql-statement-create-index.md | 88 +++++++++++++++++--- system-variables.md | 6 ++ tidb-configuration-file.md | 11 +-- 3 files changed, 84 insertions(+), 21 deletions(-) diff --git a/sql-statements/sql-statement-create-index.md b/sql-statements/sql-statement-create-index.md index f59aba3a17dd6..788f1a9fe79a8 100644 --- a/sql-statements/sql-statement-create-index.md +++ b/sql-statements/sql-statement-create-index.md @@ -104,44 +104,108 @@ Query OK, 0 rows affected (0.31 sec) ## Expression index +In some scenarios, query conditions run a filtering process based on a certain expression. In this case, the query performance is relatively poor because general indexes can not take effect in those scenarios, so the query can only be completed by traversing the entire table. Expression indexes are the special index that can be built on an expression. Once an expression index is built, you can use the index for the expression-based query, which significantly speeds up the query performance. + +If you want to build the indexes based on `col1+cols2`, run the following SQL statement as an example: + +{{< copyable "sql" >}} + +```sql +CREATE INDEX idx1 ON t1 ((col1 + col2)); +``` + +Or you can run the following alternative statement: + +{{< copyable "sql" >}} + +```sql +ALTER TABLE t1 ADD INDEX idx1((col1 + col2)); +``` + +You can also specify the expression index as you build the table: + +{{< copyable "sql" >}} + +```sql +CREATE TABLE t1(col1 char(10), col2 char(10), key index((col1 + col2))); +``` + +The method of deleting an expression index is the same as that of deleting a general index: + +{{< copyable "sql" >}} + +```sql +DROP INDEX idx1 ON t1; +``` + > **Note:** > -> Expression index is still an experimental feature. It is **NOT** recommended that you use it in the production environment. +> An expression index can not be a primary key. +> +> An expression with expression indexes can not contain the following: +> +> - volatile functions, such as `rand()`, `now()`, and so on. +> - system variables and user variables. +> - a subquery. +> - `AUTO_INCREMENT` column. However, there is one exception: you can remove this restriction by setting the value of `tidb_enable_auto_increment_in_generated` (system variable) to `true`. +> - window functions. +> - the ROW functions, such as `create table t (j json, key k (((j,j))));`. +> - aggregate functions. +> +> An expression index takes names implicitly by setting `_V$_{index_name}_{index_offset}`. Suppose you try to set a new expression index's name that a column has taken when creating an expression index. In that case, an error returns in this process. Besides, if you add a column with an existing name, the same error returns. +> +> The number of parameters of the functions that are in an expression with expression indexes must be correct. -To use this feature, make the following setting in [TiDB Configuration File](/tidb-configuration-file.md#allow-expression-index-new-in-v400): +When the expression in a query statement matches the expression in an expression index, the optimizer can choose to use the expression index for the query. In some cases, the optimizer may not select an expression index depending on statistics. At that time, you can force the optimizer to select an expression index by specifying optimizer hints. + +For example, suppose you build the expression index `idx` on the expression `lower(col1)`: + +If the results of the query statement are the same expressions, you can use expression indexes. Take the following statements as an example: {{< copyable "sql" >}} ```sql -allow-expression-index = true +SELECT lower(col1) FROM t; ``` -TiDB can build indexes not only on one or more columns in a table, but also on an expression. When queries involve expressions, expression indexes can speed up those queries. +If the same expressions are included in the filtering conditions, you can use expression indexes. Take the following statements as an example: -Take the following query as an example: +{{< copyable "sql" >}} + +```sql +SELECT * FROM t WHERE lower(col1) = "a"; +SELECT * FROM t WHERE lower(col1) > "a"; +SELECT * FROM t WHERE lower(col1) BETWEEN "a" AND "b"; +SELECT * FROM t WHERE lower(col1) in ("a", "b"); +SELECT * FROM t WHERE lower(col1) > "a" AND lower(col1) < "b"; +SELECT * FROM t WHERE lower(col1) > "b" OR lower(col1) < "a"; +``` + +When the queries are sorted by the same expressions, you can use expression indexes. Take the following statements as an example: {{< copyable "sql" >}} ```sql -SELECT * FROM t WHERE lower(name) = "pingcap"; +SELECT * FROM t ORDER BY lower(col1); ``` -If the following expression index is built, you can use the index to speed up the above query: +If the same expressions are included in the aggregate (`GROUP BY`) functions, you can use expression indexes. Take the following statements as an example: {{< copyable "sql" >}} ```sql -CREATE INDEX idx ON t ((lower(name))); +SELECT max(lower(col1)) FROM t； +SELECT min(col1) FROM t GROUP BY lower(col1); ``` +To see the expression corresponding to the expression index, run `show index`, or check the system table `information_schema.tidb_indexes` and the table `information_schema.STATISTICS`. The Expression in the output indicates the corresponded expression. As for the indexes that are not expression indexes, the column shows `NULL`. + The cost of maintaining an expression index is higher than that of maintaining other indexes, because the value of the expression needs to be calculated whenever a row is inserted or updated. The value of the expression is already stored in the index, so this value does not require recalculation when the optimizer selects the expression index. Therefore, when the query performance outweighs the insert and update performance, you can consider indexing the expressions. Expression indexes have the same syntax and limitations as in MySQL. They are implemented by building indexes on generated virtual columns that are invisible, so the supported expressions inherit all [limitations of virtual generated columns](/generated-columns.md#limitations). -Currently, the optimizer can use the indexed expressions when the expressions are only in the `FIELD` clause, `WHERE` clause, and `ORDER BY` clause. The `GROUP BY` clause will be supported in future updates. - ## Invisible index Invisible indexes are indexes that are ignored by the query optimizer: @@ -155,13 +219,15 @@ For details, see [`ALTER INDEX`](/sql-statements/sql-statement-alter-index.md). ## Associated session variables -The global variables associated with the `CREATE INDEX` statement are `tidb_ddl_reorg_worker_cnt`, `tidb_ddl_reorg_batch_size` and `tidb_ddl_reorg_priority`. Refer to [system variables](/system-variables.md#tidb_ddl_reorg_worker_cnt) for details. +The global variables associated with the `CREATE INDEX` statement are `tidb_ddl_reorg_worker_cnt`, `tidb_ddl_reorg_batch_size`, `tidb_ddl_reorg_priority`, and `tidb_enable_auto_increment_in_generated`. Refer to [system variables](/system-variables.md#tidb_ddl_reorg_worker_cnt) for details. ## MySQL compatibility * `FULLTEXT`, `HASH` and `SPATIAL` indexes are not supported. * Descending indexes are not supported (similar to MySQL 5.7). * Adding the primary key of the `CLUSTERED` type to a table is not supported. For more details about the primary key of the `CLUSTERED` type, refer to [clustered index](/clustered-indexes.md). +* Expression indexes are incompatible with the views. When you query contents via a view, the expression index can not be used at the same time. +* There are compatibility issues between expression indexes and bindings. When an expression in an expression index has constants, the binding created for the corresponding query expands its scope. For example, suppose the expression in the expression index is `a+1`, and the corresponding query condition is `a+1 > 2`. In this case, the binding is `a+? > ?`, which means that the queries such as `a+2 > 2` are also forced to select expression indexes. It results in a poor execution plan. Besides, this also affects the baseline capturing and baseline evolution belonging to SQL Plan Management. ## See also diff --git a/system-variables.md b/system-variables.md index 8a31979541327..bb9b9ce216564 100644 --- a/system-variables.md +++ b/system-variables.md @@ -501,6 +501,12 @@ Constraint checking is always performed in place for pessimistic transactions (d > - If you have enabled TiDB Binlog, enabling this variable cannot improve the performance. To improve the performance, it is recommended to use [TiCDC](/ticdc/ticdc-overview.md) instead. > - Enabling this parameter only means that Async Commit becomes an optional mode of transaction commit. In fact, the most suitable mode of transaction commit is determined by TiDB. +### tidb_enable_auto_increment_in_generated + +- Scope: SESSION | GLOBAL +- Default value: `OFF` +- This variable is used to determine whether to involve the `AUTO_INCREMENT` columns when creating a generated column or an expression index. + ### tidb_enable_cascades_planner - Scope: SESSION | GLOBAL diff --git a/tidb-configuration-file.md b/tidb-configuration-file.md index c856bb976f7b1..bdc3f1254b15f 100644 --- a/tidb-configuration-file.md +++ b/tidb-configuration-file.md @@ -636,13 +636,4 @@ For pessimistic transaction usage, refer to [TiDB Pessimistic Transaction Mode]( + The maximum number of deadlock events that can be recorded in the [`INFORMATION_SCHEMA.DEADLOCKS`](/information-schema/information-schema-deadlocks.md) table of a single TiDB server. If this table is in full volume and an additional deadlock event occurs, the earliest record in the table will be removed to make place for the newest error. + Default value: `10` + Minimum value: `0` -+ Maximum value: `10000` - -## experimental - -The `experimental` section, introduced in v3.1.0, describes configurations related to the experimental features of TiDB. - -### `allow-expression-index` New in v4.0.0 - -- Determines whether to create the expression index. -- Default value: `false` ++ Maximum value: `10000` \ No newline at end of file From acba8e9bda07082ae5a0fede8237634b5a59df37 Mon Sep 17 00:00:00 2001 From: en-jin19 Date: Wed, 18 Aug 2021 10:11:49 +0200 Subject: [PATCH 02/10] revise translations --- sql-statements/sql-statement-create-index.md | 2 +- system-variables.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/sql-statements/sql-statement-create-index.md b/sql-statements/sql-statement-create-index.md index 788f1a9fe79a8..7fa749c96aa47 100644 --- a/sql-statements/sql-statement-create-index.md +++ b/sql-statements/sql-statement-create-index.md @@ -227,7 +227,7 @@ The global variables associated with the `CREATE INDEX` statement are `tidb_ddl_ * Descending indexes are not supported (similar to MySQL 5.7). * Adding the primary key of the `CLUSTERED` type to a table is not supported. For more details about the primary key of the `CLUSTERED` type, refer to [clustered index](/clustered-indexes.md). * Expression indexes are incompatible with the views. When you query contents via a view, the expression index can not be used at the same time. -* There are compatibility issues between expression indexes and bindings. When an expression in an expression index has constants, the binding created for the corresponding query expands its scope. For example, suppose the expression in the expression index is `a+1`, and the corresponding query condition is `a+1 > 2`. In this case, the binding is `a+? > ?`, which means that the queries such as `a+2 > 2` are also forced to select expression indexes. It results in a poor execution plan. Besides, this also affects the baseline capturing and baseline evolution belonging to SQL Plan Management. +* There are compatibility issues between expression indexes and bindings. When an expression in an expression index has constants, the binding created for the corresponding query expands its scope. For example, suppose the expression in the expression index is `a+1`, and the corresponding query condition is `a+1 > 2`. In this case, the binding is `a+? > ?`, which means that the queries such as `a+2 > 2` are also forced to use the expression index. It results in a poor execution plan. Besides, this also affects the baseline capturing and baseline evolution belonging to SQL Plan Management. ## See also diff --git a/system-variables.md b/system-variables.md index bb9b9ce216564..b4ef0a1174c3d 100644 --- a/system-variables.md +++ b/system-variables.md @@ -505,7 +505,7 @@ Constraint checking is always performed in place for pessimistic transactions (d - Scope: SESSION | GLOBAL - Default value: `OFF` -- This variable is used to determine whether to involve the `AUTO_INCREMENT` columns when creating a generated column or an expression index. +- This variable is used to determine whether can include the `AUTO_INCREMENT` columns when creating a generated column or an expression index. ### tidb_enable_cascades_planner From 1165f31fcf0882ebb49ce5cbd04c7dda20ce6170 Mon Sep 17 00:00:00 2001 From: en-jin19 Date: Wed, 18 Aug 2021 17:23:32 +0200 Subject: [PATCH 03/10] reviews a part of translation --- sql-statements/sql-statement-create-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-statements/sql-statement-create-index.md b/sql-statements/sql-statement-create-index.md index 7fa749c96aa47..c15c573bcaf61 100644 --- a/sql-statements/sql-statement-create-index.md +++ b/sql-statements/sql-statement-create-index.md @@ -226,7 +226,7 @@ The global variables associated with the `CREATE INDEX` statement are `tidb_ddl_ * `FULLTEXT`, `HASH` and `SPATIAL` indexes are not supported. * Descending indexes are not supported (similar to MySQL 5.7). * Adding the primary key of the `CLUSTERED` type to a table is not supported. For more details about the primary key of the `CLUSTERED` type, refer to [clustered index](/clustered-indexes.md). -* Expression indexes are incompatible with the views. When you query contents via a view, the expression index can not be used at the same time. +* Expression indexes are incompatible with views. When you query contents by using views, the expression index can not be used at the same time. * There are compatibility issues between expression indexes and bindings. When an expression in an expression index has constants, the binding created for the corresponding query expands its scope. For example, suppose the expression in the expression index is `a+1`, and the corresponding query condition is `a+1 > 2`. In this case, the binding is `a+? > ?`, which means that the queries such as `a+2 > 2` are also forced to use the expression index. It results in a poor execution plan. Besides, this also affects the baseline capturing and baseline evolution belonging to SQL Plan Management. ## See also From dd6e1a73a546160c96e6dd3c98122adcc8cb32d3 Mon Sep 17 00:00:00 2001 From: Enwei Date: Thu, 19 Aug 2021 13:16:07 +0200 Subject: [PATCH 04/10] Apply suggestions from code review Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- sql-statements/sql-statement-create-index.md | 40 ++++++++++---------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/sql-statements/sql-statement-create-index.md b/sql-statements/sql-statement-create-index.md index c15c573bcaf61..c22b7e52f3da9 100644 --- a/sql-statements/sql-statement-create-index.md +++ b/sql-statements/sql-statement-create-index.md @@ -104,9 +104,9 @@ Query OK, 0 rows affected (0.31 sec) ## Expression index -In some scenarios, query conditions run a filtering process based on a certain expression. In this case, the query performance is relatively poor because general indexes can not take effect in those scenarios, so the query can only be completed by traversing the entire table. Expression indexes are the special index that can be built on an expression. Once an expression index is built, you can use the index for the expression-based query, which significantly speeds up the query performance. +In some scenarios, query conditions run a filtering process based on a certain expression. In these scenarios, the query performance is relatively poor because ordinary indexes cannot take effect, the query can only be executed by scanning the entire table. The expression index is a type of special index that can be built on an expression. Once an expression index is built, TiDB can use the index for the expression-based query, which significantly improves the query performance. -If you want to build the indexes based on `col1+cols2`, run the following SQL statement as an example: +For example, if you want to build an index based on `col1+cols2`, execute the following SQL statement: {{< copyable "sql" >}} @@ -114,7 +114,7 @@ If you want to build the indexes based on `col1+cols2`, run the following SQL st CREATE INDEX idx1 ON t1 ((col1 + col2)); ``` -Or you can run the following alternative statement: +Or you can execute the following equivalent statement: {{< copyable "sql" >}} @@ -122,7 +122,7 @@ Or you can run the following alternative statement: ALTER TABLE t1 ADD INDEX idx1((col1 + col2)); ``` -You can also specify the expression index as you build the table: +You can also specify the expression index when you build the table: {{< copyable "sql" >}} @@ -130,7 +130,7 @@ You can also specify the expression index as you build the table: CREATE TABLE t1(col1 char(10), col2 char(10), key index((col1 + col2))); ``` -The method of deleting an expression index is the same as that of deleting a general index: +You can drop an expression index in the same way as dropping an ordinary index: {{< copyable "sql" >}} @@ -140,27 +140,27 @@ DROP INDEX idx1 ON t1; > **Note:** > -> An expression index can not be a primary key. +> An expression index cannot be created on a primary key. > -> An expression with expression indexes can not contain the following: +> The expression in an expression index cannot contain the following content: > > - volatile functions, such as `rand()`, `now()`, and so on. > - system variables and user variables. -> - a subquery. +> -Subqueries. > - `AUTO_INCREMENT` column. However, there is one exception: you can remove this restriction by setting the value of `tidb_enable_auto_increment_in_generated` (system variable) to `true`. > - window functions. -> - the ROW functions, such as `create table t (j json, key k (((j,j))));`. +> - ROW functions, such as `create table t (j json, key k (((j,j))));`. > - aggregate functions. > -> An expression index takes names implicitly by setting `_V$_{index_name}_{index_offset}`. Suppose you try to set a new expression index's name that a column has taken when creating an expression index. In that case, an error returns in this process. Besides, if you add a column with an existing name, the same error returns. +> An expression index implicitly takes up a name (for example, `_V$_{index_name}_{index_offset}`). If you try to create a new expression index with the name that a column has already had, an error occurs. In addition, if you add a new column with the same name, an error also occurs. > -> The number of parameters of the functions that are in an expression with expression indexes must be correct. +> Make sure that the number of function parameters in the expression of an expression index is correct. -When the expression in a query statement matches the expression in an expression index, the optimizer can choose to use the expression index for the query. In some cases, the optimizer may not select an expression index depending on statistics. At that time, you can force the optimizer to select an expression index by specifying optimizer hints. +When the expression in a query statement matches the expression in an expression index, the optimizer can choose the expression index for the query. In some cases, the optimizer might not choose an expression index depending on statistics. In this situation, you can force the optimizer to select an expression index by using optimizer hints. -For example, suppose you build the expression index `idx` on the expression `lower(col1)`: +In the following examples, suppose that you build the expression index `idx` on the expression `lower(col1)`: -If the results of the query statement are the same expressions, you can use expression indexes. Take the following statements as an example: +If the results of the query statement are the same expressions, the expression index applies. Take the following statement as an example: {{< copyable "sql" >}} @@ -168,7 +168,7 @@ If the results of the query statement are the same expressions, you can use expr SELECT lower(col1) FROM t; ``` -If the same expressions are included in the filtering conditions, you can use expression indexes. Take the following statements as an example: +If the same expression is included in the filtering conditions, the expression index applies. Take the following statements as an example: {{< copyable "sql" >}} @@ -181,7 +181,7 @@ SELECT * FROM t WHERE lower(col1) > "a" AND lower(col1) < "b"; SELECT * FROM t WHERE lower(col1) > "b" OR lower(col1) < "a"; ``` -When the queries are sorted by the same expressions, you can use expression indexes. Take the following statements as an example: +When the queries are sorted by the same expression, the expression index applies. Take the following statement as an example: {{< copyable "sql" >}} @@ -189,7 +189,7 @@ When the queries are sorted by the same expressions, you can use expression inde SELECT * FROM t ORDER BY lower(col1); ``` -If the same expressions are included in the aggregate (`GROUP BY`) functions, you can use expression indexes. Take the following statements as an example: +If the same expression is included in the aggregate (`GROUP BY`) functions, the expression index applies. Take the following statement as an example: {{< copyable "sql" >}} @@ -198,7 +198,7 @@ SELECT max(lower(col1)) FROM t； SELECT min(col1) FROM t GROUP BY lower(col1); ``` -To see the expression corresponding to the expression index, run `show index`, or check the system table `information_schema.tidb_indexes` and the table `information_schema.STATISTICS`. The Expression in the output indicates the corresponded expression. As for the indexes that are not expression indexes, the column shows `NULL`. +To see the expression corresponding to the expression index, execute `show index`, or check the system tables `information_schema.tidb_indexes` and the table `information_schema.STATISTICS`. The `Expression` column in the output indicates the corresponded expression. For the non-expression indexes, the column shows `NULL`. The cost of maintaining an expression index is higher than that of maintaining other indexes, because the value of the expression needs to be calculated whenever a row is inserted or updated. The value of the expression is already stored in the index, so this value does not require recalculation when the optimizer selects the expression index. @@ -226,8 +226,8 @@ The global variables associated with the `CREATE INDEX` statement are `tidb_ddl_ * `FULLTEXT`, `HASH` and `SPATIAL` indexes are not supported. * Descending indexes are not supported (similar to MySQL 5.7). * Adding the primary key of the `CLUSTERED` type to a table is not supported. For more details about the primary key of the `CLUSTERED` type, refer to [clustered index](/clustered-indexes.md). -* Expression indexes are incompatible with views. When you query contents by using views, the expression index can not be used at the same time. -* There are compatibility issues between expression indexes and bindings. When an expression in an expression index has constants, the binding created for the corresponding query expands its scope. For example, suppose the expression in the expression index is `a+1`, and the corresponding query condition is `a+1 > 2`. In this case, the binding is `a+? > ?`, which means that the queries such as `a+2 > 2` are also forced to use the expression index. It results in a poor execution plan. Besides, this also affects the baseline capturing and baseline evolution belonging to SQL Plan Management. +* Expression indexes are incompatible with views. When a query is executed using a view, the expression index cannot be used at the same time. +* Expression indexes have compatibility issues with bindings. When the expression of an expression index has a constant, the binding created for the corresponding query expands its scope. For example, suppose that the expression in the expression index is `a+1`, and the corresponding query condition is `a+1 > 2`. In this case, the created binding is `a+? > ?`, which means that the query with the condition such as `a+2 > 2` is also forced to use the expression index and results in a poor execution plan. In addition, this also affects the baseline capturing and baseline evolution in SQL Plan Management (SPM). ## See also From f2b7ba414ba1500ae35b810ba4feb901f85e9fe6 Mon Sep 17 00:00:00 2001 From: Enwei Date: Thu, 19 Aug 2021 13:16:51 +0200 Subject: [PATCH 05/10] Apply suggestions from code review --- sql-statements/sql-statement-create-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-statements/sql-statement-create-index.md b/sql-statements/sql-statement-create-index.md index c22b7e52f3da9..08d7b350180d0 100644 --- a/sql-statements/sql-statement-create-index.md +++ b/sql-statements/sql-statement-create-index.md @@ -144,7 +144,7 @@ DROP INDEX idx1 ON t1; > > The expression in an expression index cannot contain the following content: > -> - volatile functions, such as `rand()`, `now()`, and so on. +> - Volatile functions, such as `rand()`, `now()`, and so on. > - system variables and user variables. > -Subqueries. > - `AUTO_INCREMENT` column. However, there is one exception: you can remove this restriction by setting the value of `tidb_enable_auto_increment_in_generated` (system variable) to `true`. From 1547a70ed67e8320e8ef72bce00b23bb66e06bb4 Mon Sep 17 00:00:00 2001 From: Enwei Date: Fri, 20 Aug 2021 08:19:13 +0200 Subject: [PATCH 06/10] Apply suggestions and remove "build" Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- sql-statements/sql-statement-create-index.md | 26 ++++++++++---------- system-variables.md | 2 +- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/sql-statements/sql-statement-create-index.md b/sql-statements/sql-statement-create-index.md index 08d7b350180d0..b407d75e2ccfd 100644 --- a/sql-statements/sql-statement-create-index.md +++ b/sql-statements/sql-statement-create-index.md @@ -104,9 +104,9 @@ Query OK, 0 rows affected (0.31 sec) ## Expression index -In some scenarios, query conditions run a filtering process based on a certain expression. In these scenarios, the query performance is relatively poor because ordinary indexes cannot take effect, the query can only be executed by scanning the entire table. The expression index is a type of special index that can be built on an expression. Once an expression index is built, TiDB can use the index for the expression-based query, which significantly improves the query performance. +In some scenarios, query conditions run a filtering process based on a certain expression. In these scenarios, the query performance is relatively poor because ordinary indexes cannot take effect, the query can only be executed by scanning the entire table. The expression index is a type of special index that can be created on an expression. Once an expression index is created, TiDB can use the index for the expression-based query, which significantly improves the query performance. -For example, if you want to build an index based on `col1+cols2`, execute the following SQL statement: +For example, if you want to create an index based on `col1+cols2`, execute the following SQL statement: {{< copyable "sql" >}} @@ -122,7 +122,7 @@ Or you can execute the following equivalent statement: ALTER TABLE t1 ADD INDEX idx1((col1 + col2)); ``` -You can also specify the expression index when you build the table: +You can also specify the expression index when you create the table: {{< copyable "sql" >}} @@ -144,13 +144,13 @@ DROP INDEX idx1 ON t1; > > The expression in an expression index cannot contain the following content: > -> - Volatile functions, such as `rand()`, `now()`, and so on. -> - system variables and user variables. -> -Subqueries. -> - `AUTO_INCREMENT` column. However, there is one exception: you can remove this restriction by setting the value of `tidb_enable_auto_increment_in_generated` (system variable) to `true`. -> - window functions. +> - Volatile functions, such as `rand()`, `now()`. +> - System variables and user variables. +> - Subqueries. +> - `AUTO_INCREMENT` column. There is an exception: you can remove this restriction by setting the value of `tidb_enable_auto_increment_in_generated` (system variable) to `true`. +> - Window functions. > - ROW functions, such as `create table t (j json, key k (((j,j))));`. -> - aggregate functions. +> - Aggregate functions. > > An expression index implicitly takes up a name (for example, `_V$_{index_name}_{index_offset}`). If you try to create a new expression index with the name that a column has already had, an error occurs. In addition, if you add a new column with the same name, an error also occurs. > @@ -158,7 +158,7 @@ DROP INDEX idx1 ON t1; When the expression in a query statement matches the expression in an expression index, the optimizer can choose the expression index for the query. In some cases, the optimizer might not choose an expression index depending on statistics. In this situation, you can force the optimizer to select an expression index by using optimizer hints. -In the following examples, suppose that you build the expression index `idx` on the expression `lower(col1)`: +In the following examples, suppose that you create the expression index `idx` on the expression `lower(col1)`: If the results of the query statement are the same expressions, the expression index applies. Take the following statement as an example: @@ -189,7 +189,7 @@ When the queries are sorted by the same expression, the expression index applies SELECT * FROM t ORDER BY lower(col1); ``` -If the same expression is included in the aggregate (`GROUP BY`) functions, the expression index applies. Take the following statement as an example: +If the same expression is included in the aggregate (`GROUP BY`) functions, the expression index applies. Take the following statements as an example: {{< copyable "sql" >}} @@ -204,7 +204,7 @@ The cost of maintaining an expression index is higher than that of maintaining o Therefore, when the query performance outweighs the insert and update performance, you can consider indexing the expressions. -Expression indexes have the same syntax and limitations as in MySQL. They are implemented by building indexes on generated virtual columns that are invisible, so the supported expressions inherit all [limitations of virtual generated columns](/generated-columns.md#limitations). +Expression indexes have the same syntax and limitations as in MySQL. They are implemented by creating indexes on generated virtual columns that are invisible, so the supported expressions inherit all [limitations of virtual generated columns](/generated-columns.md#limitations). ## Invisible index @@ -227,7 +227,7 @@ The global variables associated with the `CREATE INDEX` statement are `tidb_ddl_ * Descending indexes are not supported (similar to MySQL 5.7). * Adding the primary key of the `CLUSTERED` type to a table is not supported. For more details about the primary key of the `CLUSTERED` type, refer to [clustered index](/clustered-indexes.md). * Expression indexes are incompatible with views. When a query is executed using a view, the expression index cannot be used at the same time. -* Expression indexes have compatibility issues with bindings. When the expression of an expression index has a constant, the binding created for the corresponding query expands its scope. For example, suppose that the expression in the expression index is `a+1`, and the corresponding query condition is `a+1 > 2`. In this case, the created binding is `a+? > ?`, which means that the query with the condition such as `a+2 > 2` is also forced to use the expression index and results in a poor execution plan. In addition, this also affects the baseline capturing and baseline evolution in SQL Plan Management (SPM). +* Expression indexes have compatibility issues with bindings. When the expression of an expression index has a constant, the binding created for the corresponding query expands its scope. For example, suppose that the expression in the expression index is `a+1`, and the corresponding query condition is `a+1 > 2`. In this case, the created binding is `a+? > ?`, which means that the query with the condition such as `a+2 > 2` is also forced to use the expression index and results in a poor execution plan. In addition, this also affects the baseline capturing and baseline evolution in SQL Plan Management (SPM). ## See also diff --git a/system-variables.md b/system-variables.md index b4ef0a1174c3d..67e6635c220df 100644 --- a/system-variables.md +++ b/system-variables.md @@ -505,7 +505,7 @@ Constraint checking is always performed in place for pessimistic transactions (d - Scope: SESSION | GLOBAL - Default value: `OFF` -- This variable is used to determine whether can include the `AUTO_INCREMENT` columns when creating a generated column or an expression index. +- This variable is used to determine whether to include the `AUTO_INCREMENT` columns when creating a generated column or an expression index. ### tidb_enable_cascades_planner From 1defadf1de5e3165ce370acf2919948159294a9a Mon Sep 17 00:00:00 2001 From: Enwei Date: Fri, 20 Aug 2021 08:27:33 +0200 Subject: [PATCH 07/10] Update sql-statements/sql-statement-create-index.md --- sql-statements/sql-statement-create-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-statements/sql-statement-create-index.md b/sql-statements/sql-statement-create-index.md index b407d75e2ccfd..0b01258cf7998 100644 --- a/sql-statements/sql-statement-create-index.md +++ b/sql-statements/sql-statement-create-index.md @@ -144,7 +144,7 @@ DROP INDEX idx1 ON t1; > > The expression in an expression index cannot contain the following content: > -> - Volatile functions, such as `rand()`, `now()`. +> - Volatile functions, such as `rand()` and `now()`. > - System variables and user variables. > - Subqueries. > - `AUTO_INCREMENT` column. There is an exception: you can remove this restriction by setting the value of `tidb_enable_auto_increment_in_generated` (system variable) to `true`. From 8a4987954b4a7f0431e84ae1477f838c5f9fcb38 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 20 Aug 2021 15:46:04 +0800 Subject: [PATCH 08/10] Apply suggestions from code review --- sql-statements/sql-statement-create-index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sql-statements/sql-statement-create-index.md b/sql-statements/sql-statement-create-index.md index 0b01258cf7998..0ce58d8a219a4 100644 --- a/sql-statements/sql-statement-create-index.md +++ b/sql-statements/sql-statement-create-index.md @@ -104,7 +104,7 @@ Query OK, 0 rows affected (0.31 sec) ## Expression index -In some scenarios, query conditions run a filtering process based on a certain expression. In these scenarios, the query performance is relatively poor because ordinary indexes cannot take effect, the query can only be executed by scanning the entire table. The expression index is a type of special index that can be created on an expression. Once an expression index is created, TiDB can use the index for the expression-based query, which significantly improves the query performance. +In some scenarios, the filtering condition of a query is based on a certain expression. In these scenarios, the query performance is relatively poor because ordinary indexes cannot take effect, the query can only be executed by scanning the entire table. The expression index is a type of special index that can be created on an expression. Once an expression index is created, TiDB can use the index for the expression-based query, which significantly improves the query performance. For example, if you want to create an index based on `col1+cols2`, execute the following SQL statement: @@ -147,7 +147,7 @@ DROP INDEX idx1 ON t1; > - Volatile functions, such as `rand()` and `now()`. > - System variables and user variables. > - Subqueries. -> - `AUTO_INCREMENT` column. There is an exception: you can remove this restriction by setting the value of `tidb_enable_auto_increment_in_generated` (system variable) to `true`. +> - `AUTO_INCREMENT` column. You can remove this restriction by setting the value of `tidb_enable_auto_increment_in_generated` (system variable) to `true`. > - Window functions. > - ROW functions, such as `create table t (j json, key k (((j,j))));`. > - Aggregate functions. From cdcd00b0363550e748ad01ca16a04c9826772580 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 20 Aug 2021 15:51:37 +0800 Subject: [PATCH 09/10] Update sql-statements/sql-statement-create-index.md --- sql-statements/sql-statement-create-index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sql-statements/sql-statement-create-index.md b/sql-statements/sql-statement-create-index.md index 0ce58d8a219a4..4cd0b4611172e 100644 --- a/sql-statements/sql-statement-create-index.md +++ b/sql-statements/sql-statement-create-index.md @@ -155,6 +155,8 @@ DROP INDEX idx1 ON t1; > An expression index implicitly takes up a name (for example, `_V$_{index_name}_{index_offset}`). If you try to create a new expression index with the name that a column has already had, an error occurs. In addition, if you add a new column with the same name, an error also occurs. > > Make sure that the number of function parameters in the expression of an expression index is correct. +> +> When the expression of the index contains a string-related function, affected by the returned type and the length, creating the expression index might fail. In this situation, you can use the `cast()` function to explicitly specify the returned type and the length. When the expression in a query statement matches the expression in an expression index, the optimizer can choose the expression index for the query. In some cases, the optimizer might not choose an expression index depending on statistics. In this situation, you can force the optimizer to select an expression index by using optimizer hints. From b4ff50ef5fa527197834f62e1ed99d44ffa60613 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 20 Aug 2021 15:52:10 +0800 Subject: [PATCH 10/10] Update sql-statements/sql-statement-create-index.md --- sql-statements/sql-statement-create-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-statements/sql-statement-create-index.md b/sql-statements/sql-statement-create-index.md index 4cd0b4611172e..e1d59f081089f 100644 --- a/sql-statements/sql-statement-create-index.md +++ b/sql-statements/sql-statement-create-index.md @@ -156,7 +156,7 @@ DROP INDEX idx1 ON t1; > > Make sure that the number of function parameters in the expression of an expression index is correct. > -> When the expression of the index contains a string-related function, affected by the returned type and the length, creating the expression index might fail. In this situation, you can use the `cast()` function to explicitly specify the returned type and the length. +> When the expression of an index contains a string-related function, affected by the returned type and the length, creating the expression index might fail. In this situation, you can use the `cast()` function to explicitly specify the returned type and the length. When the expression in a query statement matches the expression in an expression index, the optimizer can choose the expression index for the query. In some cases, the optimizer might not choose an expression index depending on statistics. In this situation, you can force the optimizer to select an expression index by using optimizer hints.