From cf2ac749d44ab6ff20cd710a691999eb9f1710ae Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 19 Jan 2022 14:02:11 +0800 Subject: [PATCH 01/27] Add TiDB v5.4 release notes --- TOC.md | 2 + _index.md | 2 +- releases/release-5.4.0.md | 430 +++++++++++++++++++++++++++++++++++ releases/release-timeline.md | 1 + 4 files changed, 434 insertions(+), 1 deletion(-) create mode 100644 releases/release-5.4.0.md diff --git a/TOC.md b/TOC.md index 809d639762f97..c4116f291de93 100644 --- a/TOC.md +++ b/TOC.md @@ -672,6 +672,8 @@ - Release Notes - [All Releases](/releases/release-notes.md) - [Release Timeline](/releases/release-timeline.md) + - v5.4 + - [5.4.0](/releases/release-5.4.0.md) - v5.3 - [5.3.0](/releases/release-5.3.0.md) - v5.2 diff --git a/_index.md b/_index.md index 0ca603dd913c8..4a9fba50ac97e 100644 --- a/_index.md +++ b/_index.md @@ -15,7 +15,7 @@ Designed for the cloud, TiDB provides flexible scalability, reliability and secu - [TiDB Introduction](/overview.md) - [Basic Features](/basic-features.md) -- [TiDB 5.3 Release Notes](/releases/release-5.3.0.md) +- [TiDB 5.4 Release Notes](/releases/release-5.4.0.md) - [TiDB Release Timeline](/releases/release-timeline.md) - [Compatibility with MySQL](/mysql-compatibility.md) - [Usage Limitations](/tidb-limitations.md) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md new file mode 100644 index 0000000000000..0cd90e7b224fe --- /dev/null +++ b/releases/release-5.4.0.md @@ -0,0 +1,430 @@ +--- +title: TiDB 5.4 Release Notes +--- + +# TiDB 5.4 Release Notes + +Release date：February xx, 2022 + +TiDB version: 5.4.0 + +In v5.4, the key new features or improvements are as follows: + +- Support the GBK character set +- Support using Index Merge to access data, which merges the filtering results of indexes on multiple columns +- Support reading stale data using a session variable +- Support persisting the configuration for collecting statistics +- Support using Raft Engine as the log storage engine of TiKV (experimental) +- Optimize the impact of backup on the cluster +- Support using Azure Blob storage as the target storage for backup +- Continuouly improve the stability and performance of TiFlash and the MPP engine +- Add a switch in TiDB Lightning to determine whether to allow importing an existing table +- Optimize the Continuous Profiling feature (experimental) +- TiSpark supports user identification and authentication + +## Compatibility changes + +> **Note:** +> +> When upgrading from an earlier TiDB version to v5.3.0, if you want to know the compatibility change notes of all intermediate versions, you can check the [Release Notes](/releases/release-notes.md) of the corresponding version. + +### System variables + +| Variable name | Change type | Description | +| :---------- | :----------- | :----------- | +| [`tidb_backoff_lock_fast`](/system-variables.md#tidb_backoff_lock_fast) | Modified | The default value is changed from `100` to `10`. | +| [`tidb_enable_column_tracking`](/system-variables.md#tidb_enable_column_tracking-new-in-v50) | Newly added | Controls whether allow TiDB to collect `PREDICATE COLUMNS`. The default value is `OFF`. | +| [`tidb_enable_index_merge`](/system-variables.md#tidb_enable_index_merge-new-in-v40) | Modified | The default value is changed from `OFF` to `ON`.
| +| [`tidb_enable_paging`](/system-variables.md#tidb_enable_paging-new-in-v540) | Newly added | This variable controls whether to use the method of paging to send coprocessor requests in `IndexLookUp` operator. The default value is `OFF`.
For read queries that use `IndexLookup` and `Limit` and that `Limit` cannot be pushed down to `IndexScan`, there might be high latency for the read queries and high CPU usage for TiKV's `unified read pool`. In such cases, because the `Limit` operator only requires a small set of data, if you set `tidb_enable_paging` to `ON`, TiDB processes less data, which reduces query latency and resource consumption. | +| [`tidb_enable_top_sql`](/system-variables.md#tidb_enable_top_sql-new-in-v540) | Newly added | Controls whether to enable the Top SQL feature. The default value is `OFF`. | +| [`tidb_persist_analyze_options`](/system-variables.md#tidb_persist_analyze_options-new-in-v540) | Newly added | Controls whether to enable the [ANALYZE configuration persistence](/statistics.md#persist-analyze-configurations) feature. The default value is `OFF`. | +| [`tidb_read_staleness`](/system-variables.md#tidb_read_staleness-new-in-v540) | Newly added | Controls the range of historical data that can be read in the current session. The default value is `0`.| +| [`tidb_regard_null_as_point `](/system-variables.md#tidb_regard_null_as_point-new-in-v540) | Newly added | Controls whether the optimizer can treat `null` as a point value and uses it as a prefix condition to access the index 用于控制优化器是否可以把 null 值当做点值并作为前缀条件来访问索引 | +| [`tidb_stats_load_sync_wait`](/system-variables.md#tidb_stats_load_sync_wait-new-in-v540) | Newly added | This variable controls whether to enable the synchronously loading statistics feature. (The default value `0` means that the feature is disabled and that the statistics is asynchronously loaded.) When the feature is enabled, this variable controls the maximum time that SQL optimization can wait for synchronously loading statistics before timeout. | +| [`tidb_stats_load_pseudo_timeout`](/system-variables.md#tidb_stats_load_pseudo_timeout-new-in-v540) | Newly added | This variable controls when synchronously loading statistics reaches timeout, whether SQL fails (`OFF`) or falls back to using pseudo statistics. The default value is `ON`. | +| [`tidb_store_limit`](/system-variables.md#tidb_store_limit-new-i-v304-and-v40) | Modified | Before v5.4.0, this variabe can be configured at instance level and globally. Starting from v5.4.0, this variable only supports global configuration. | + +### Configuration file parameters + +| Configuration file | Configuration | Change type | Description | +| :---------- | :----------- | :----------- | :----------- | +| TiDB | [`stats-load-concurrency`](/tidb-configuration-file.md#stats-load-concurrency-new-in-v540) | Newly added | Controls the maximum number of columns that the TiDB synchronously loading statistics feature can process concurrently. The default value is `5` | +| TiDB | [`stats-load-queue-size`](/tidb-configuration-file.md#stats-load-queue-size-new-in-v540) | Newly added | Controls the maximum number of column requests that the TiDB synchronously loading statistics feature can cache. The default value is`1000`. | +| TiKV | [`snap-generator-pool-size`](/tidb-configuration-file.md#snap-generator-pool-size) | Newly added | The size of `snap-generator` thread pool. The default value is `2`. | +| TiKV | `log.file.max-size`、`log.file.max-days`、`log.file.max-backups` | Newly added | For details, see [TiKV Configuration File - log.file](/tikv-configuration-file.md#logfile-new-in-v540). | +| TiKV | `raft-engine` | Newly added | Includes `enable`, `dir`, `batch-compression-threshold`, `bytes-per-sync`, `target-file-size`, `purge-threshold`, `recovery-mode`, `recovery-read-block-size`, `recovery-read-block-size`, `recovery-threads`. For details, see [TiKV Configuration File - Raft-engine](/tikv-configuration-file.md#raft-engine).| +| TiKV | [`backup.enable-auto-tune`](/tikv-configuration-file.md#enable-auto-tune-new-in-v540) | Newly added | In v5.3.0, the default value is `false`. Since v5.4.0, the default value is changed to `true`. This parameter controls whether to limit the resources used by backup tasks to reduce the impact on the cluster when the cluster resource utilization is high. In the default configuration, the speed of backup task might slow down. | +| TiKV | `log-level`, `log-format`, `log-file`, `log-rotation-size` | Modified | The names of TiKV log parameters are replaced with the names that are same as TiDB log parameters, which are `log.level`, `log.format`, `log.file.filename`, `log.enable-timestamp`. If you only set the old parameters, and their values are set to non-default values, the old parameters remain compatible with the new parameters. If both old and new parameters are set, the new parameters take effect. For details, see [TiKV Configuration File - log](/tikv-configuration-file.md#log-new-in-v540). | +| TiKV | `log-rotation-timespan` | Deleted | The timespan between log rotations. When this timespan passes, log files are rotated, that is, a timestamp is appended to the file name of the current log file, and a new file is created. | +| TiKV | `allow-remove-leader` | Deleted | Determines whether to allow deleting the main switch. | +| TiKV | `raft-msg-flush-interval` | Deleted | Determines the interval at which Raft messages are sent in batches. The Raft messages in batches are sent at every interval specified by this configuration item. | +| PD | [`log.level`](/pd-configuration-file.md#level) | Modified | The default value is changed from "INFO" to "info", guaranteed to be case in-sensitive. | +| TiFlash | [`profile.default.enable_elastic_threadpool`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Newly added | Determines whether to enable or disable the elastic thread pool function. Enabling this configuration item can significantly improve TiFlash CPU utilization in high concurrency scenarios. The default value is `false`. | +| TiFlash | [`storage.format_version`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Newly added | Specifies the version of DTFile. The default value is `2`, under which hashes are embedded in the data file. You can also set the value to `3`. When it is `3`, the data file contains metadata and token data checksum, and supports multiple hash algorithms. | +| TiFlash | [`logger.count`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Modified | The default value is changed to `10`. | +| TiFlash | [`status.metrics_port`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Modified | The default value is changed to `8234`. | +| TiFlash | [`raftstore.apply-pool-size`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) | Newly added | The allowable number of threads in the pool that flushes Raft data to storage. The default value is `4`. | +| TiFlash | [`raftstore.store-pool-size`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) | Newly added | The allowable number of threads that process Raft, which is the size of the Raftstore thread pool. The default value is `4`. | +| TiDB Data Migration (DM) | [`collation_compatible`](/dm/task-configuration-file-full.md##task-configuration-file-template-advanced) | Newly added | The mode to sync the default collation in `CREATE` SQL statements. The supported values are "loose" (by default) or "strict". | +| TiCDC | `max-message-bytes` | Modified| Change the default value of `max-message-bytes` in Kafka sink to `104857601` (10MB) | +| TiCDC | `partition-num` | Modified | Change the default value of `partition-num` in Kafka Sink from `4` to `3`. It makes TiCDC send messages to Kafaka partitions more evenly. | +| TiDB Lightning | `meta-schema-name` | Modified | Specifies the schema name for the metadata in the target TiDB. From v5.4.0, this schema is created only if you have enabled [parallel import](/tidb-lightning/tidb-lightning-distributed-import) (the corresponding parameter is `tikv-importer.incremental-import = true`). | +| TiDB Lightning | `task-info-schema-name` | Newly added | Specifies the name of the database where duplicated data is stored when Lightning detects conflicts. By default, the value is "lightning_task_info". Specify this parameter only if you have enabled the "duplicate-resolution" feature. | +| TiDB Lightning | `incremental-import` | Newly added | Whether to allow importing data to tables where data already exists. The default value is `false`. | + + +### Others + +- TiDB Dashboard, by default, does not permit using `root` + empty password for login. + Since v5.4.0, it is recommended to use `start --initial` when you use TiUP to start a TiDB cluster. Then, a random password is generated for the `root` account to log into TiDB Dashboard. + +- An interface is added between TiDB and PD. When using the `information_schema.TIDB_HOT_REGIONS_HISTORY` system table, TiDB needs to use the PD in a matching version. +- TiDB Server, PD Server, and TiKV Server start using a unified naming method for the log-related parameters to manage log names, output formats, the rules of rotation and expiration. For details, see [TiKV configuration file - log](/tikv-configuration-file.md#log-new-in-v540). +- Since v5.4.0, if you create a SQL binding for an execution plan that has been cached via Plan Cache, the binding invalidates the plan already cached for the corresponding query. Before v5.4.0, the new binding does not affect any execution plans that have been already cached. + +## New features + +### SQL + +- **TiDB supports GBK since v5.4.0** + + Before v5.4.0, TiDB supports `ascii`, `binary`, `latin1`, `utf8`, and `utf8mb4` character sets. + + To better support Chinese users, TiDB supports the GBK character set since v5.4.0. After enabling the [`new_collations_enabled_on_first_bootstrap`](/tidb-configuration-file.md#new_collations_enabled_on_first_bootstrap) option in the TiDB configuration file when initializing a TiDB cluster for the first time, the TiDB GBK character set supports both `gbk_bin` and `gbk_chinese_ci` collations. + + When using the GBK character set, you need to pay attention to the compatibility restrictions. For details, see [Character Set and Collation - GBK](/character-set-gbk.md). + +### Security + +- **TiSpark supports user authentication and authorization** + + Since TiSpark 2.5.0, TiSpark supports both database user authentication and read/write authorization at a database or table level. After enabling this feature, you can prevent the business from running unauthorized batch tasks such as draws to obtain data, which improves the stability and data security of online clusters. + + This feature is disabled by default. When it is enabled, if a user operating through TiSpark does not have the needed permissions, the user gets an exception from TiSpark. + + [User document](/tispark-overview.md#security) + +### Performance + +- **Continue improving the stability and performance of the column-wise storage engine and the MPP computing engine for TiFlash** + + - Support more functions to be pushed down to the MPP engine: + - String functions: `LPAD()`, `RPAD()`, `STRCMP()` + - Date functions: `ADDDATE()`, `DATE_ADD()`, `DATE_SUB()`, `SUBDATE()`, `QUARTER()` + - Introduce the elastic thread pool function to improve resource utilization (experimental feature) + - Accelerate row-column data conversion when replicating data from TiKV. This improves overall replication performance by 50%. + - Improve TiFlash performance and stability by adjusting the default value of some configuration items, yielding up to 20% performance improvement in simple queries per table. + + User documents: [Supported push-down calculations](/tiflash/use-tiflash.md#supported-push-down-calculations), [Configure the tiflash.toml file](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) + +- **Read expired data within a specified time range through a session variable** + + TiDB is a multi-replica distributed database that uses the Raft protocol. In the high concurrency and high throughput business scenarios, TiDB can make read performance scalable through follower nodes and build a read-write separation architecture. + + For different business scenarios, the follower provides two read modes: strongly consistent read and weakly consistent expired read. The strongly consistent read mode is suitable for use in business scenarios with strict data real-time requirements. However, when this mode is used, especially in the geo-distributed deployment, there will be latency issue due to the data replication latency and the reduced throughput of the leader and the follower. + + For the business scenarios that do not require high real-time performance, the expired read mode can be used. This mode can reduce latency and improve throughput. TiDB currently supports the expired read by displaying read-only transactions or SQL statements. Both methods support reading the historical data of a specific point in time or within a specified time range. For details, please refer to [Read Historical Data through session variable `tidb_snapshot`](/read-historical-data.md). + + Since v5.4.0, TiDB supports reading historical data within a specified time range a through session variable, which further improves usability, reduces the difficulty of development, and meets the business requirements of low latency and high throughput of data access in quasi-real-time scenarios. You can set the variable as following example: + + ```sql + set @@tidb_replica_read=leader_and_follower + set @@tidb_read_staleness="-5" + ``` + + By this setting, TiDB can select the nearest leader or follower node and read the latest historical data within 5 seconds. + + [User document](/read-historical-data.md) + +- **GA for Index Merge** + + _Index Merge_ is introduced in TiDB v4.0 an experimental feature to access tables. This method greatly accelerates condition filtering when a query requires scanning of multiple columns of data. Take the following query as an example. In the `WHERE` statement, the filtering conditions connected by `OR` have their respective indexes in columns _key1_ and _key2_. Based on the indexes in the two columns, the Index Merge feature filters and merges the query results, and returns the merged result. + + ```sql + SELECT * FROM table WHERE key1 <= 100 OR key2 = 200; + ``` + + Before TiDB v4.0, a query in a table uses only one index and cannot use multiple indexes for filtering. If you want to query multiple columns of data, you can enable _Inex Merge_ to get the exact query result in short time by using the indexes in individual columns. _Inex Merge_ avoids unnecessary full table scans and does not require establishing large number of composite indexes. + + In v5.4.0, _Inex Merge_ is a GA feature. However, you still need to pay attention to the following restrictions: + + - _Inex Merge_ supports only disjunctive normal form (X1 ⋁ X2 ⋁ …Xn). That is, to use this feature, filtering conditions in a `WHERE` clause should be connected by `OR`. + + - This feature is enabled by default for TiDB clusters of v5.4.0 or later. For v5.4.0 or later TiDB clusters upgraded from versions earlier than v5.4.0, this feature inherits the setting before the upgrade and you can change the setting as required (in TiDB clusters earlier than v4.0, this feature does not exist and is disabled by default). + + [User document](/explain-index-merge.md) + +- **Add Raft Engine (experimental)** + + Support using [Raft Engine](https://github.com/tikv/raft-engine) as the log storage engine in TiKV. Compared with RocksDB, Raft Engine can reduce TiKV I/O write traffic by up to 40% and CPU usage by 10%, while improving foreground throughput by about 5% and reducing tail latency by 20% under certain loads. In addition, Raft Engine improves the efficiency of log recycling and fixes the issue of log accumulation in extreme conditions. + + Raft Engine is still an experimental feature and is disabled by default. Note that the data format of Raft Engine in v5.4.0 is not compatible with previous versions. Before upgrading or downgrading the cluster, you need to make sure that Raft Engine on all TiKV nodes is disabled. It is recommended to use Raft Engine only in v5.4.0 or a later version. + +- **Support collecting statistics for `PREDICATE COLUMNS` (experimental)** + + In most cases, when executing SQL statements, the optimizer only uses statistics of some columns (such as columns in the `WHERE`, `JOIN`, `ORDER BY`, and `GROUP BY` statements). These used columns are called `PREDICATE COLUMNS`. + + Since v5.4.0, you can set the value of the [`tidb_enable_column_tracking`](/system-variables.md#tidb_enable_column_tracking-introduced-from-v540-version) system variable to `ON` to enable TiDB to collect `PREDICATE COLUMNS`. + + After the setting, TiDB writes the `PREDICATE COLUMNS` information to the `mysql.column_stats_usage` system table every 100 * [`stats-lease`](/tidb-configuration-file.md#stats-lease). When the query pattern of your business is stable, you can use the `ANALYZE TABLE TableName PREDICATE COLUMNS` syntax to collect statistics on the `PREDICATE COLUMNS` columns only, which can greatly reduce the overhead of collecting statistics. + + [User document](/statistics.md#collect-statistics-for-some-columns) + +- **Support synchronously loading statistics (experimental)** + + Since v5.4.0, TiDB introduces the synchronously loading statistics feature. The feature is disabled by default. After enabling the feature, TiDB can synchronously load statistics with a large space consumption (such as histogram, TopN, and Count-Min Sketch statistics) into memory when you execute SQL statements, which improves the completeness of statistics for SQL optimization. + + [[User document](/statistics.md#load-statistics) + +### Stability + +- **Support persisting ANALYZE configurations** + + Statistics are one type of the basic information that the optimizer refers to when generating execution plans. The accuracy of the statistics directly affects whether the generated execution plans are reasonable. To ensure the accuracy of the statistics, sometimes it is necessary to set different collection configurations for different tables, partitions, and indexes. + + Since v5.4.0, TiDB supports persisting some `ANALYZE` configurations. With this feature, the existing configurations can be easily reused for future statistics collection. + + The `ANALYZE` configuration persistence feature is enabled by default (the system variable `tidb_analyze_version` is `2` and `tidb_persist_analyze_options` is `ON` by default). You can use this feature to record the persistence configurations specified in the `ANALYZE` statement when executing the statement manually. Once recorded, the next time TiDB automatically updates statistics or you manually collect statistics without specifying these configuration, TiDB will collect statistics according to the recorded configurations. + + [User document](/statistics.md#persist-analyze-configurations) + +## High availability and disaster recovery + +- **Reduce the impact of backup tasks on the cluster** + + Backup & Restore (BR) introduces the auto-tune feature (enabled by default). This feature monitors the cluster resource usage and adjusts the number of threads used by the backup tasks to reduce the impact of backup tasks on the cluster. In some cases, if you increase the cluster resources for backup and enable the auto-tune feature, it is possible to limit the impact of backup tasks on the cluster to 10% or less. + + [User document](/br/br-auto-tune.md) + +- **Support Azure Blob Storage as a target storage for backup** + + Backup & Restore (BR) supports Azure Blob Storage as a remote target storage for backup. If you deploy TiDB in Azure Cloud, you can back up the cluster data to the Azure Blob Storage service. + + [User document](/br/backup-and-restore-azblob.md) + +### Data migration + +- **TiDB Lightning introduces a new feature to determine whether to allow importing data to tables with data** + + TiDB Lightning introduces a new feature `incremental-import`. It determines whether to allow importing data to tables with data. The default value is `false`. When using parallel import mode, you must set it to `true`. + + [User document](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) + +- **TiDB Lightning introduces the schema name that stores the meta information for parallel import** + + TiDB Lightning introduces `meta-schema-name`. In parallel import mode, this parameter specifies the schema name that stores the meta information for each TiDB Lightning instance in the target cluster. By default, the value is "lightning_metadata". The value set for this parameter must be the same for each TiDB Lightning instance that participates in the same parallel import; otherwise, the correctness of the imported data can not be ensured. + + [User document](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) + +- **TiDB Lightning introduces duplicate resolution** + + In Local-backend mode, TiDB Lightning outputs duplicated data before the data import is completed, and then removes that duplicated data from the database. You can resolve the duplicated data after the import is completed and select suitable data to insert according to business rules. It is recommended to clean upstream data sources based on duplicated data to avoid data inconsistency caused by duplicated data encountered in the subsequent incremental data migration phase. + + [User document](tidb-lightning/tidb-lightning-error-resolution.md) + +- **Optimize the usage of relay log in TiDB Data Migration (DM)** + + - Recover the `enable-relay` switch in the `source` configuration. + - Support dynamically enabling or disabling relay log in the `start-relay` or `stop-relay` command. + - Bind the status of relay log to `source`. `source` keeps its original status of being enabled or disabled after it is migrated to any DM-worker. + - Move the storage path of relay log to the DM-worker configuration file. + + [User document](/dm/relay-log.md) + +- **Optimize the processing of [collation](/character-set-and-collation.md) in DM** + + Add the `collation_compatible` configuration item. The value options are `loose` (default) and `strict`: + + - If your application does not have strict requirements on collation, and the collation of query results can be different between the upstream and downstream, you can use the default `loose` mode to avoid reporting errors. + - If your application has strict requirements on collation, and the collation must be consistent between the upstream and downstream, you can use the `strict` mode. However, if the downstream does not support the upstream's default collation, the data replication might report errors. + + [User document](/dm/task-configuration-file-full.md#task-configuration-file-template-advanced) + +- **Optimize `transfer source` in DM to support smooth execution of replication tasks** + + When the DM-worker nodes have unbalanced load, the `transfer source` command can be used to manually transfer the configuration of a `source` to another load. After the optimization, the `transfer source` command simplifies the manual operation. You can smoothly transfer the source instead of pausing all related tasks; DM completes other operations internally. + +- **DM OpenAPI becomes generally available (GA)** + + DM supports daily management via API, including adding data sources and managing tasks. In v5.4.0, DM OpenAPI becomes GA. + + [User document](/dm/dm-open-api.md) + +### Diagnostic efficiency + +- **Top SQL (experimental feature)** + + A new experimental feature, Top SQL (turned off by default), is introduced to help you easily find queries that consume more resources. + + [User document](/dashboard/top-sql.md) + +### TiDB data share subscription + +- **Optimize the impact of TiCDC on clusters** + + Significantly reduces the performance impact on TiDB clusters when you use TiCDC. In the test environment, the performance impact of TiCDC on TiDB can be reduced to less than 5%. + +### Deployment and maintenance + +- **Enhance Continuous Profiling (experimental feature)** + + - More components supported: Besides TiDB, PD, and TiKV, TiDB v5.4.0 also supports CPU profiling of TiFlash. + - More forms of profiling display: Supports display of CPU Profiling and Goroutine results on flame charts. + - More deployment environments supported: You can enable Continuous Profiling on clusters deployed using TiDB Operator. + + Continuous Profiling is disabled by default and can be enabled on TiDB Dashboard. + + Continuous Profiling is available for clusters deployed or upgraded using TiUP of v1.9.0 or later or TiDB Operator of v1.3.0 or later. + + [User document](/dashboard/continuous-profiling.md) + +## Improvements + ++ TiDB + + - Add a new system variable `tidb_enable_paging. For read requests in which `IndexLookUp` and `Limit` are used and `Limit` cannot be pushed down to `IndexScan`, setting this variable to `ON` can significantly reduce the latency and the resource consumption [#30578](https://github.com/pingcap/tidb/issues/30578) + - Support the `ADMIN {SESSION | INSTANCE | GLOBAL} PLAN_CACHE` syntax to clear the cached query plan [#30370](https://github.com/pingcap/tidb/pull/30370) + ++ TiKV + + - Coprocessor supports paging API to process requests in a stream-like way [#11448](https://github.com/tikv/tikv/issues/11448) + - Support `read-through-lock` so that read operations do not need to wait for secondary locks to be resolved [#11402](https://github.com/tikv/tikv/issues/11402) + - Add a disk protection mechanism to avoid panic caused by disk space drainage [#10537](https://github.com/tikv/tikv/issues/10537) + - Support archiving and rotating logs [#11651](https://github.com/tikv/tikv/issues/11651) + - Reduce the system call by the Raft client and increase CPU efficiency [#11309](https://github.com/tikv/tikv/issues/11309) + - Coprocessor supports pushing down substring to TiKV [#11495](https://github.com/tikv/tikv/issues/11495) + - Improve the scan performance by skip reading locks in the Read Committed isolation level [#11485](https://github.com/tikv/tikv/issues/11485) + - Reduce the default thread pool size used by backup operations and limit the use of thread pool when the stress is high [#11000](https://github.com/tikv/tikv/issues/11000) + - Support dynamically adjusting the sizes of Apply thread pool and Store thread pool [#11159](https://github.com/tikv/tikv/issues/11159) + - Support configuring the size of the `snap-generator` thread pool [#11247](https://github.com/tikv/tikv/issues/11247) + - Optimize the issue of global lock race that occurs when there are many files with frequent reads and writes [#250](https://github.com/tikv/rocksdb/pull/250) + ++ PD + + - Record the historic hotspot information by default [#25281](https://github.com/pingcap/tidb/issues/25281) + - Add signature for the HTTP component to identify the request source [#4490](https://github.com/tikv/pd/issues/4490) + - Update TiDB Dashboard to v2021.12.31 [#4257](https://github.com/tikv/pd/issues/4257) + ++ TiFlash + + - Optimize the communication of local operators + - Increase the non-temporary thread count of gRPC to avoid the frequent creation or destroy of threads + ++ Tools + + + Backup & Restore (BR) + + - Add a validity check for the key when BR performs encrypted backup [#29794](https://github.com/pingcap/tidb/issues/29794) + + + TiCDC + + - Reduce the count of "EventFeed retry rate limited" logs [#4006](https://github.com/pingcap/tiflow/issues/4006) + - Reduce the replication latency when replicating many tables [#3900](https://github.com/pingcap/tiflow/issues/3900) + - Reduce the time for the KV client to recover when a TiKV store is down [#3191](https://github.com/pingcap/tiflow/issues/3191) + + + TiDB Data Migration (DM) + + - Lower the usage rate of CPU when relay is enabled [#2214](https://github.com/pingcap/dm/issues/2214) + + + TiDB Lightning + + - Use optimistic transactions to write data to improve performance in TiDB-backend mode [#30953](https://github.com/pingcap/tidb/pull/30953) + + + Dumpling + + - Improve compatibility when Dumpling checks the data version [#29500](https://github.com/pingcap/tidb/pull/29500) + - Add a default collation when dumping `CREATE DATABASE` and `CREATE TABLE` [#3420](https://github.com/pingcap/tiflow/issues/3420) + +## Bug fixes + ++ TiDB + + - Fix the issue of the `tidb_analyze_version` value change that occurs when upgrading the cluster from v4.x to v5.x [#25422](https://github.com/pingcap/tidb/issues/25422) + - Fix the issue of wrong result that occurs when using different collations in a subquery [#30748](https://github.com/pingcap/tidb/issues/30748) + - Fix the issue that the result of `concat(ifnull(time(3))` in TiDB is different from that in MySQL [#29498](https://github.com/pingcap/tidb/issues/29498) + - Fix the issue of potential data index inconsistency in optimistic transaction mode [#30410](https://github.com/pingcap/tidb/issues/30410) + - Fix the issue that the query execution plan of IndexMerge is wrong when an expression cannot be pushed down to TiKV [#30200](https://github.com/pingcap/tidb/issues/30200) + - Fix the issue that concurrent column type change causes inconsistency between the schema and the data [#31048](https://github.com/pingcap/tidb/issues/31048) + - Fix the issue that the IndexMerge query result is wrong that occurs when there is a subquery [#30913](https://github.com/pingcap/tidb/issues/30913) + - Fix the panic issue that occurs when the FetchSize is set too large in the client [#30896](https://github.com/pingcap/tidb/issues/30896) + - Fix the issue that LEFT JOIN might be mistakenly converted to INNER JOIN [#20510](https://github.com/pingcap/tidb/issues/20510) + - Fix the issue that panic might occur when the `CASE-WHEN` expression and collation are used together [#30245](https://github.com/pingcap/tidb/issues/30245) + - Fix the issue of wrong query result that occurs when the `IN` value contains a binary constant [#31261](https://github.com/pingcap/tidb/issues/31261) + - Fix the issue of wrong query result that occurs when CTE has a subquery [#31255](https://github.com/pingcap/tidb/issues/31255) + - Fix the issue that executing the `INSERT ... SELECT ... ON DUPLICATE KEY UPDATE` statement get panic [#28078](https://github.com/pingcap/tidb/issues/28078) + - Fix the issue that INDEX HASH JOIN returns the `send on closed channel` error [#31129](https://github.com/pingcap/tidb/issues/31129) + ++ TiKV + + - Fix the issue that the MVCC deletion records are not cleared by GC [#11217](https://github.com/tikv/tikv/issues/11217) + - Fix the issue that retrying prewrite requests in the pessimistic transaction mode might cause the risk of data inconsistency in rare cases [#11187](https://github.com/tikv/tikv/issues/11187) + - Fix the issue that GC scan causes memory overflow [#11410](https://github.com/tikv/tikv/issues/11410) + - Fix the issue that RocksDB flush or compaction causes panic when the disk capacity is full [#11224](https://github.com/tikv/tikv/issues/11224) + ++ PD + + - Fix the issue that Region statistics are not affected by `flow-round-by-digit` [#4295](https://github.com/tikv/pd/issues/4295) + - Fix the issue that the scheduling operator cannot fail fast because the target store is down [#3353](https://github.com/tikv/pd/issues/3353) + - Fix the issue that Regions on offline stores cannot be merged [#4119](https://github.com/tikv/pd/issues/4119) + - Fix the issue that the cold hotspot data cannot be deleted from the hotspot statistics [#4390](https://github.com/tikv/pd/issues/4390) + ++ TiFlash + + - Fix the issue that the `Tree struct based executor must have executor id` error is returned in the query result + - Fix the issue that the `Illegal column type UInt32 for the second argument of function tidbRoundWithFrac` error is returned in the query result + - Fix the issue that the `Unexpected type of column: Nullable(Nothing)` error is returned in the query result + - Fix the issue that the `source region at right may be considered at left when merging` error is returned in the query result + - Fix the issue that collation does not take effect on the `nullable` type + - Fix the issue that `coalesce` mistakenly deletes the `nullable` identifier from the result columns + - Fix the issue that TiFlash might panic when a MPP query is stopped + - Fix the issue that queries with the `where ` clause return wrong result + - Fix the potential issue of data inconsistency that might occur when setting the column type of an integer primary key to a larger range + - Fix the issue that when an input time is earlier than 1970-01-01 00:00:01 UTC, the behavior of `unix_timestamp` is inconsistent with that of TiDB or MySQL + - Fix the issue that TiFlash might return the `EstablishMPPConnection` error after it is retarted + - Fix the issue that the `CastStringAsDecimal` bahavior is inconsitent in TiFlash and in TiDB/TiKV + - Fix the issue that the `DB::Exception: Encode type of coprocessor response is not CHBlock` error is returned in the query result + - Fix the issue that the `castStringAsReal` bahavior is inconsitent in TiFlash and in TiDB/TiKV + - Fix the issue that the returned result of the `date_add_string_xxx` function in TiFlash is inconsistent with that in MySQL + ++ Tools + + + Backup & Restore (BR) + + - Fix the potential issue that Region distribution might be uneven after a restore operation is finished [#30425](https://github.com/pingcap/tidb/issues/30425) + - Fix the issue that `'/'` cannot be specified in endpoint when `minio` is used as the backup storage [#30104](https://github.com/pingcap/tidb/issues/30104) + - Fix the issue that system tables cannot be restored because concurrently backing up system tables makes the table name fail to update [#29710](https://github.com/pingcap/tidb/issues/29710) + + + TiCDC + + - Fix the issue that replication cannot be performed when `min.insync.replicas` is smaller than `replication-factor` [#3994](https://github.com/pingcap/tiflow/issues/3994) + - Fix the issue that the `cached region` monitoring metric is negative [#4300](https://github.com/pingcap/tiflow/issues/4300) + - Fix the issue that `mq sink write row` does not have monitoring data [#3431](https://github.com/pingcap/tiflow/issues/3431) + - Fix the compatibility issue of `sql mode` [#3810](https://github.com/pingcap/tiflow/issues/3810) + - Fix the potential panic issue that occurs when a replication task is removed [#3128](https://github.com/pingcap/tiflow/issues/3128) + - Fix the issue of panic and data inconsistency that occurs when outputting the default column value [#3929](https://github.com/pingcap/tiflow/issues/3929) + - Fix the issue that default values cannot be replicated [#3793](https://github.com/pingcap/tiflow/issues/3793) + - Fix the potential issue that the deadlock causes a replication task to gets stuck [#4055](https://github.com/pingcap/tiflow/issues/4055) + - Fix the issue that no log is output when the disk is fully written [#3362](https://github.com/pingcap/tiflow/issues/3362) + - Fix the issue that special comments in DDL statements cause the replication task to stop [#3755](https://github.com/pingcap/tiflow/issues/3755) + - Fix the issue that the service cannot be started because of a timezone issue in the RHEL release [#3584](https://github.com/pingcap/tiflow/issues/3584) + - Fix the issue of potential data loss caused by inaccurate checkpoint [#3545](https://github.com/pingcap/tiflow/issues/3545) + - Fix the OOM issue in the container environment [#1798](https://github.com/pingcap/tiflow/issues/1798) + - Fix the issue of replication stop caused by the incorrect configuration of `config.Metadata.Timeout` [#3352](https://github.com/pingcap/tiflow/issues/3352) + + + TiDB Data Migration (DM) + + - Fix the issue that the `CREATE VIEW` statement interrupts data replication [#4173](https://github.com/pingcap/tiflow/issues/4173) + - Fix the issue the schema needs to be reset after a DDL statement is skipped [#4177](https://github.com/pingcap/tiflow/issues/4177) + - Fix the issue that the table checkpoint is not updated in time after a DDL statement is skipped [#4184](https://github.com/pingcap/tiflow/issues/4184) + - Fix a compatibility issue of the TiDB version with the parser version [#4298](https://github.com/pingcap/tiflow/issues/4298) + - Fix the issue that syncer metrics are updated only when querying the status [#4281](https://github.com/pingcap/tiflow/issues/4281) + + + TiDB Lightning + + - Fix the issue of wrong import result that occurs when TiDB Lightning does not have the priviledge to access the `mysql.tidb` table [#31088](https://github.com/pingcap/tidb/issues/31088) + - Fix the issue that some checks are skipped when TiDB Lightning is restarted [#30772](https://github.com/pingcap/tidb/issues/30772) + - Fix the issue that TiDB Lighting fails to report the error when the S3 path does not exist [#30674](https://github.com/pingcap/tidb/pull/30674) + + + TiDB Binlog + + - Fix the issue that Drainer fails because it is incompatible with the `CREATE PLACEMENT POLICY` statement [#1118](https://github.com/pingcap/tidb-binlog/issues/1118) diff --git a/releases/release-timeline.md b/releases/release-timeline.md index e1ef19e83a511..4d10acbad4b56 100644 --- a/releases/release-timeline.md +++ b/releases/release-timeline.md @@ -9,6 +9,7 @@ This document shows all the released TiDB versions in reverse chronological orde | Version | Release Date | | :--- | :--- | +| [5.4.0](/releases/release-5.4.0.md) | 2022-x-x | | [5.0.6](/releases/release-5.0.6.md) | 2021-12-31 | | [4.0.16](/releases/release-4.0.16.md) | 2021-12-17 | | [5.1.3](/releases/release-5.1.3.md) | 2021-12-03 | From f3b931995cab3f4e09171be6ad523cab65eb3c13 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 19 Jan 2022 14:04:26 +0800 Subject: [PATCH 02/27] Update releases/release-5.4.0.md --- releases/release-5.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index 0cd90e7b224fe..a99f579a6a443 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -26,7 +26,7 @@ In v5.4, the key new features or improvements are as follows: > **Note:** > -> When upgrading from an earlier TiDB version to v5.3.0, if you want to know the compatibility change notes of all intermediate versions, you can check the [Release Notes](/releases/release-notes.md) of the corresponding version. +> When upgrading from an earlier TiDB version to v5.4.0, if you want to know the compatibility change notes of all intermediate versions, you can check the [Release Notes](/releases/release-notes.md) of the corresponding version. ### System variables From 8dcccdf950083bb13aede065f529e0388fbc367c Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Wed, 19 Jan 2022 15:05:04 +0800 Subject: [PATCH 03/27] Apply suggestions from code review --- releases/release-5.4.0.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index a99f579a6a443..ea7f1e8ca5370 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -138,13 +138,13 @@ In v5.4, the key new features or improvements are as follows: - **GA for Index Merge** - _Index Merge_ is introduced in TiDB v4.0 an experimental feature to access tables. This method greatly accelerates condition filtering when a query requires scanning of multiple columns of data. Take the following query as an example. In the `WHERE` statement, the filtering conditions connected by `OR` have their respective indexes in columns _key1_ and _key2_. Based on the indexes in the two columns, the Index Merge feature filters and merges the query results, and returns the merged result. + _Index Merge_ is introduced in TiDB v4.0 as an experimental feature to access tables. This method greatly accelerates condition filtering when a query requires scanning of multiple columns of data. Take the following query as an example. In the `WHERE` statement, the filtering conditions connected by `OR` have their respective indexes in columns _key1_ and _key2_. Based on the indexes in the two columns, the Index Merge feature filters and merges the query results, and returns the merged result. ```sql SELECT * FROM table WHERE key1 <= 100 OR key2 = 200; ``` - Before TiDB v4.0, a query in a table uses only one index and cannot use multiple indexes for filtering. If you want to query multiple columns of data, you can enable _Inex Merge_ to get the exact query result in short time by using the indexes in individual columns. _Inex Merge_ avoids unnecessary full table scans and does not require establishing large number of composite indexes. + Before TiDB v4.0, a query in a table uses only one not multiple indexes for filtering. If you want to query multiple columns of data, you can enable _Inex Merge_ to get the exact query result in short time by using the indexes in individual columns. _Inex Merge_ avoids unnecessary full table scans and does not require establishing large number of composite indexes. In v5.4.0, _Inex Merge_ is a GA feature. However, you still need to pay attention to the following restrictions: @@ -269,12 +269,12 @@ In v5.4, the key new features or improvements are as follows: - **Enhance Continuous Profiling (experimental feature)** - More components supported: Besides TiDB, PD, and TiKV, TiDB v5.4.0 also supports CPU profiling of TiFlash. - - More forms of profiling display: Supports display of CPU Profiling and Goroutine results on flame charts. - - More deployment environments supported: You can enable Continuous Profiling on clusters deployed using TiDB Operator. + - More forms of profiling display: Supports display of CPU profiling and Goroutine results on flame charts. + - More deployment environments supported: Continuous Profiling is applicable to clusters deployed using TiDB Operator. Continuous Profiling is disabled by default and can be enabled on TiDB Dashboard. - Continuous Profiling is available for clusters deployed or upgraded using TiUP of v1.9.0 or later or TiDB Operator of v1.3.0 or later. + Continuous Profiling is available to clusters deployed or upgraded using TiUP of v1.9.0 or later or TiDB Operator of v1.3.0 or later. [User document](/dashboard/continuous-profiling.md) From e168a9f47b90c1ffedc0ff47e065213a92c938d5 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Thu, 20 Jan 2022 14:42:02 +0800 Subject: [PATCH 04/27] update basic feature and experimental features --- basic-features.md | 265 ++++++++++++++++++++------------------- experimental-features.md | 14 ++- 2 files changed, 142 insertions(+), 137 deletions(-) diff --git a/basic-features.md b/basic-features.md index 11c87251820a5..d243f5088e486 100644 --- a/basic-features.md +++ b/basic-features.md @@ -9,168 +9,169 @@ This document lists the features supported in each TiDB version. Note that suppo ## Data types, functions, and operators -| Data types, functions, and operators | 5.3 | 5.2 | 5.1 | 5.0 | 4.0 | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Numeric types](/data-type-numeric.md) | Y | Y | Y | Y | Y | -| [Date and time types](/data-type-date-and-time.md) | Y | Y | Y | Y | Y | -| [String types](/data-type-string.md) | Y | Y | Y | Y | Y | -| [JSON type](/data-type-json.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Control flow functions](/functions-and-operators/control-flow-functions.md) | Y | Y | Y | Y | Y | -| [String functions](/functions-and-operators/string-functions.md) | Y | Y | Y | Y | Y | -| [Numeric functions and operators](/functions-and-operators/numeric-functions-and-operators.md) | Y | Y | Y | Y | Y | -| [Date and time functions](/functions-and-operators/date-and-time-functions.md) | Y | Y | Y | Y | Y | -| [Bit functions and operators](/functions-and-operators/bit-functions-and-operators.md) | Y | Y | Y | Y | Y | -| [Cast functions and operators](/functions-and-operators/cast-functions-and-operators.md) | Y | Y | Y | Y | Y | -| [Encryption and compression functions](/functions-and-operators/encryption-and-compression-functions.md) | Y | Y | Y | Y | Y | -| [Information functions](/functions-and-operators/information-functions.md) | Y | Y | Y | Y | Y | -| [JSON functions](/functions-and-operators/json-functions.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Aggregation functions](/functions-and-operators/aggregate-group-by-functions.md) | Y | Y | Y | Y | Y | -| [Window functions](/functions-and-operators/window-functions.md) | Y | Y | Y | Y | Y | -| [Miscellaneous functions](/functions-and-operators/miscellaneous-functions.md) | Y | Y | Y | Y | Y | -| [Operators](/functions-and-operators/operators.md) | Y | Y | Y | Y | Y | -| [Character sets and collations](/character-set-and-collation.md) [^1] | Y | Y | Y | Y | Y | +| Data types, functions, and operators | **5.4** | 5.3 | 5.2 | 5.1 | 5.0 | 4.0 | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| [Numeric types](/data-type-numeric.md) | Y | Y | Y | Y | Y | Y | +| [Date and time types](/data-type-date-and-time.md) | Y | Y | Y | Y | Y | Y | +| [String types](/data-type-string.md) | Y | Y | Y | Y | Y | Y | +| [JSON type](/data-type-json.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Control flow functions](/functions-and-operators/control-flow-functions.md) | Y | Y | Y | Y | Y | Y | +| [String functions](/functions-and-operators/string-functions.md) | Y | Y | Y | Y | Y | Y | +| [Numeric functions and operators](/functions-and-operators/numeric-functions-and-operators.md) | Y | Y | Y | Y | Y | Y | +| [Date and time functions](/functions-and-operators/date-and-time-functions.md) | Y | Y | Y | Y | Y | Y | +| [Bit functions and operators](/functions-and-operators/bit-functions-and-operators.md) | Y | Y | Y | Y | Y | Y | +| [Cast functions and operators](/functions-and-operators/cast-functions-and-operators.md) | Y | Y | Y | Y | Y | Y | +| [Encryption and compression functions](/functions-and-operators/encryption-and-compression-functions.md) | Y | Y | Y | Y | Y | Y | +| [Information functions](/functions-and-operators/information-functions.md) | Y | Y | Y | Y | Y | Y | +| [JSON functions](/functions-and-operators/json-functions.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Aggregation functions](/functions-and-operators/aggregate-group-by-functions.md) | Y | Y | Y | Y | Y | Y | +| [Window functions](/functions-and-operators/window-functions.md) | Y | Y | Y | Y | Y | Y | +| [Miscellaneous functions](/functions-and-operators/miscellaneous-functions.md) | Y | Y | Y | Y | Y | Y | +| [Operators](/functions-and-operators/operators.md) | Y | Y | Y | Y | Y | Y | +| [Character sets and collations](/character-set-and-collation.md) [^1] | Y | Y | Y | Y | Y | Y | ## Indexing and constraints -| Indexing and constraints | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Expression indexes](/sql-statements/sql-statement-create-index.md#expression-index) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Columnar storage (TiFlash)](/tiflash/tiflash-overview.md) | Y | Y | Y | Y | Y | -| [RocksDB engine](/storage-engine/rocksdb-overview.md) | Y | Y | Y | Y | Y | -| [Titan plugin](/storage-engine/titan-overview.md) | Y | Y | Y | Y | Y | -| [Invisible indexes](/sql-statements/sql-statement-add-index.md) | Y | Y | Y | Y | N | -| [Composite `PRIMARY KEY`](/constraints.md) | Y | Y | Y | Y | Y | -| [Unique indexes](/constraints.md) | Y | Y | Y | Y | Y | -| [Clustered index on integer `PRIMARY KEY`](/constraints.md) | Y | Y | Y | Y | Y | -| [Clustered index on composite or non-integer key](/constraints.md) | Y | Y | Y | Y | N | +| Indexing and constraints | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| [Expression indexes](/sql-statements/sql-statement-create-index.md#expression-index) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Columnar storage (TiFlash)](/tiflash/tiflash-overview.md) | Y | Y | Y | Y | Y | Y | +| [RocksDB engine](/storage-engine/rocksdb-overview.md) | Y | Y | Y | Y | Y | Y | +| [Titan plugin](/storage-engine/titan-overview.md) | Y | Y | Y | Y | Y | Y | +| [Invisible indexes](/sql-statements/sql-statement-add-index.md) | Y | Y | Y | Y | Y | N | +| [Composite `PRIMARY KEY`](/constraints.md) | Y | Y | Y | Y | Y | Y | +| [Unique indexes](/constraints.md) | Y | Y | Y | Y | Y | Y | +| [Clustered index on integer `PRIMARY KEY`](/constraints.md) | Y | Y | Y | Y | Y | Y | +| [Clustered index on composite or non-integer key](/constraints.md) | Y | Y | Y | Y | Y | N | ## SQL statements -| **SQL statements** [^2] | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| Basic `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `REPLACE` | Y | Y | Y | Y | Y | -| `INSERT ON DUPLICATE KEY UPDATE` | Y | Y | Y | Y | Y | -| `LOAD DATA INFILE` | Y | Y | Y | Y | Y | -| `SELECT INTO OUTFILE` | Y | Y | Y | Y | Y | -| `INNER JOIN`, `LEFT\|RIGHT [OUTER] JOIN` | Y | Y | Y | Y | Y | -| `UNION`, `UNION ALL` | Y | Y | Y | Y | Y | -| [`EXCEPT` and `INTERSECT` operators](/functions-and-operators/set-operators.md) | Y | Y | Y | Y | N | -| `GROUP BY`, `ORDER BY` | Y | Y | Y | Y | Y | -| [Window Functions](/functions-and-operators/window-functions.md) | Y | Y | Y | Y | Y | -| [Common Table Expressions (CTE)](/sql-statements/sql-statement-with.md) | Y | Y | Y | N | N | -| `START TRANSACTION`, `COMMIT`, `ROLLBACK` | Y | Y | Y | Y | Y | -| [`EXPLAIN`](/sql-statements/sql-statement-explain.md) | Y | Y | Y | Y | Y | -| [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md) | Y | Y | Y | Y | Y | -| [User-defined variables](/user-defined-variables.md) | Experimental | Experimental | Experimental | Experimental | Experimental | +| **SQL statements** [^2] | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| Basic `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `REPLACE` | Y | Y | Y | Y | Y | Y | +| `INSERT ON DUPLICATE KEY UPDATE` | Y | Y | Y | Y | Y | Y | +| `LOAD DATA INFILE` | Y | Y | Y | Y | Y | Y | +| `SELECT INTO OUTFILE` | Y | Y | Y | Y | Y | Y | +| `INNER JOIN`, `LEFT\|RIGHT [OUTER] JOIN` | Y | Y | Y | Y | Y | Y | +| `UNION`, `UNION ALL` | Y | Y | Y | Y | Y | Y | +| [`EXCEPT` and `INTERSECT` operators](/functions-and-operators/set-operators.md) | Y | Y | Y | Y | Y | N | +| `GROUP BY`, `ORDER BY` | Y | Y | Y | Y | Y | Y | +| [Window Functions](/functions-and-operators/window-functions.md) | Y | Y | Y | Y | Y | Y | +| [Common Table Expressions (CTE)](/sql-statements/sql-statement-with.md) | Y | Y | Y | Y | N | N | +| `START TRANSACTION`, `COMMIT`, `ROLLBACK` | Y | Y | Y | Y | Y | Y | +| [`EXPLAIN`](/sql-statements/sql-statement-explain.md) | Y | Y | Y | Y | Y | Y | +| [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md) | Y | Y | Y | Y | Y | Y | +| [User-defined variables](/user-defined-variables.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | ## Advanced SQL features -| **Advanced SQL features** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Prepared statement cache](/sql-prepare-plan-cache.md) | Y | Experimental | Experimental | Experimental | Experimental | -| [SQL plan management (SPM)](/sql-plan-management.md) | Y | Y | Y | Y | Y | -| [Coprocessor cache](/coprocessor-cache.md) | Y | Y | Y | Y | Experimental | -| [Stale Read](/stale-read.md) | Y | Y | Y | N | N | -| [Follower reads](/follower-read.md) | Y | Y | Y | Y | Y | -| [Read historical data (tidb_snapshot)](/read-historical-data.md) | Y | Y | Y | Y | Y | -| [Optimizer hints](/optimizer-hints.md) | Y | Y | Y | Y | Y | -| [MPP Exection Engine](/explain-mpp.md) | Y | Y | Y | Y | N | -| [Index Merge Join](/explain-index-merge.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Placement Rules in SQL](/placement-rules-in-sql.md) | Experimental | N | N | N | N | +| **Advanced SQL features** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| [Prepared statement cache](/sql-prepare-plan-cache.md) | Y | Y | Experimental | Experimental | Experimental | Experimental | +| [SQL plan management (SPM)](/sql-plan-management.md) | Y | Y | Y | Y | Y | Y | +| [Coprocessor cache](/coprocessor-cache.md) | Y | Y | Y | Y | Y | Experimental | +| [Stale Read](/stale-read.md) | Y | Y | Y | Y | N | N | +| [Follower reads](/follower-read.md) | Y | Y | Y | Y | Y | Y | +| [Read historical data (tidb_snapshot)](/read-historical-data.md) | Y | Y | Y | Y | Y | Y | +| [Optimizer hints](/optimizer-hints.md) | Y | Y | Y | Y | Y | Y | +| [MPP Exection Engine](/explain-mpp.md) | Y | Y | Y | Y | Y | N | +| [Index Merge Join](/explain-index-merge.md) | Y | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Placement Rules in SQL](/placement-rules-in-sql.md) | Experimental | Experimental | N | N | N | N | ## Data definition language (DDL) -| **Data definition language (DDL)** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| Basic `CREATE`, `DROP`, `ALTER`, `RENAME`, `TRUNCATE` | Y | Y | Y | Y | Y | -| [Generated columns](/generated-columns.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Views](/views.md) | Y | Y | Y | Y | Y | -| [Sequences](/sql-statements/sql-statement-create-sequence.md) | Y | Y | Y | Y | Y | -| [Auto increment](/auto-increment.md) | Y | Y | Y | Y | Y | -| [Auto random](/auto-random.md) | Y | Y | Y | Y | Y | -| [DDL algorithm assertions](/sql-statements/sql-statement-alter-table.md) | Y | Y | Y | Y | Y | -| Multi schema change: add column(s) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Change column type](/sql-statements/sql-statement-modify-column.md) | Y | Y | Y | N | N | -| [Temporary tables](/temporary-tables.md) | Y | N | N | N | N | +| **Data definition language (DDL)** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| Basic `CREATE`, `DROP`, `ALTER`, `RENAME`, `TRUNCATE` | Y | Y | Y | Y | Y | Y | +| [Generated columns](/generated-columns.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Views](/views.md) | Y | Y | Y | Y | Y | Y | +| [Sequences](/sql-statements/sql-statement-create-sequence.md) | Y | Y | Y | Y | Y | Y | +| [Auto increment](/auto-increment.md) | Y | Y | Y | Y | Y | Y | +| [Auto random](/auto-random.md) | Y | Y | Y | Y | Y | Y | +| [DDL algorithm assertions](/sql-statements/sql-statement-alter-table.md) | Y | Y | Y | Y | Y | Y | +| Multi schema change: add column(s) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Change column type](/sql-statements/sql-statement-modify-column.md) | Y | Y | Y | Y | N | N | +| [Temporary tables](/temporary-tables.md) | Y | Y | N | N | N | N | ## Transactions -| **Transactions** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Async commit](/system-variables.md#tidb_enable_async_commit-new-in-v50) | Y | Y | Y | Y | N | -| [1PC](/system-variables.md#tidb_enable_1pc-new-in-v50) | Y | Y | Y | Y | N | -| [Large transactions (10GB)](/transaction-overview.md#transaction-size-limit) | Y | Y | Y | Y | Y | -| [Pessimistic transactions](/pessimistic-transaction.md) | Y | Y | Y | Y | Y | -| [Optimistic transactions](/optimistic-transaction.md) | Y | Y | Y | Y | Y | -| [Repeatable-read isolation (snapshot isolation)](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | -| [Read-committed isolation](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | +| **Transactions** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ---- | :-----: | :-----: | :-----: | :-----: | :-----: | +| [Async commit](/system-variables.md#tidb_enable_async_commit-new-in-v50) | Y | Y | Y | Y | Y | N | +| [1PC](/system-variables.md#tidb_enable_1pc-new-in-v50) | Y | Y | Y | Y | Y | N | +| [Large transactions (10GB)](/transaction-overview.md#transaction-size-limit) | Y | Y | Y | Y | Y | Y | +| [Pessimistic transactions](/pessimistic-transaction.md) | Y | Y | Y | Y | Y | Y | +| [Optimistic transactions](/optimistic-transaction.md) | Y | Y | Y | Y | Y | Y | +| [Repeatable-read isolation (snapshot isolation)](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | Y | +| [Read-committed isolation](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | Y | ## Partitioning -| **Partitioning** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Range partitioning](/partitioned-table.md) | Y | Y | Y | Y | Y | -| [Hash partitioning](/partitioned-table.md) | Y | Y | Y | Y | Y | -| [List partitioning](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | N | -| [List COLUMNS partitioning](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | N | -| [`EXCHANGE PARTITION`](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | N | -| [Dynamic Pruning](/partitioned-table.md#dynamic-pruning-mode) | Experimental | Experimental | Experimental | N | N | +| **Partitioning** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :-----: | +| [Range partitioning](/partitioned-table.md) | Y | Y | Y | Y | Y | Y | +| [Hash partitioning](/partitioned-table.md) | Y | Y | Y | Y | Y | Y | +| [List partitioning](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | Experimental | N | +| [List COLUMNS partitioning](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | Experimental | N | +| [`EXCHANGE PARTITION`](/partitioned-table.md) | Experimental | Experimental | Experimental | Experimental | Experimental | N | +| [Dynamic Pruning](/partitioned-table.md#dynamic-pruning-mode) | Experimental | Experimental | Experimental | Experimental | N | N | ## Statistics -| **Statistics** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [CMSketch](/statistics.md) | Deprecated | Deprecated | Deprecated | Deprecated | Y | -| [Histograms](/statistics.md) | Y | Y | Y | Y | Y | -| [Extended statistics (multiple columns)](/statistics.md) | Experimental | Experimental | Experimental | Experimental | N | -| [Statistics Feedback](/statistics.md#automatic-update) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Fast Analyze](/system-variables.md#tidb_enable_fast_analyze) | Experimental | Experimental | Experimental | Experimental | Experimental | +| **Statistics** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| [CMSketch](/statistics.md) | Deprecated | Deprecated | Deprecated | Deprecated | Deprecated | Y | +| [Histograms](/statistics.md) | Y | Y | Y | Y | Y | Y | +| [Extended statistics (multiple columns)](/statistics.md) | Experimental | Experimental | Experimental | Experimental | Experimental | N | +| [Statistics Feedback](/statistics.md#automatic-update) | Deprecated | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Fast Analyze](/system-variables.md#tidb_enable_fast_analyze) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | ## Security -| **Security** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Transparent layer security (TLS)](/enable-tls-between-clients-and-servers.md) | Y | Y | Y | Y | Y | -| [Encryption at rest (TDE)](/encryption-at-rest.md) | Y | Y | Y | Y | Y | -| [Role-based authentication (RBAC)](/role-based-access-control.md) | Y | Y | Y | Y | Y | -| [Certificate-based authentication](/certificate-authentication.md) | Y | Y | Y | Y | Y | -| `caching_sha2_password` authentication | Y | Y | N | N | N | -| [MySQL compatible `GRANT` system](/privilege-management.md) | Y | Y | Y | Y | Y | -| [Dynamic Privileges](/privilege-management.md#dynamic-privileges) | Y | Y | Y | N | N | -| [Security Enhanced Mode](/system-variables.md#tidb_enable_enhanced_security) | Y | Y | Y | N | N | -| [Redacted Log Files](/log-redaction.md) | Y | Y | Y | Y | N | +| **Security** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ---- | :-----: | :-----: | :-----: | :-----: | :-----: | +| [Transparent layer security (TLS)](/enable-tls-between-clients-and-servers.md) | Y | Y | Y | Y | Y | Y | +| [Encryption at rest (TDE)](/encryption-at-rest.md) | Y | Y | Y | Y | Y | Y | +| [Role-based authentication (RBAC)](/role-based-access-control.md) | Y | Y | Y | Y | Y | Y | +| [Certificate-based authentication](/certificate-authentication.md) | Y | Y | Y | Y | Y | Y | +| `caching_sha2_password` authentication | Y | Y | Y | N | N | N | +| [MySQL compatible `GRANT` system](/privilege-management.md) | Y | Y | Y | Y | Y | Y | +| [Dynamic Privileges](/privilege-management.md#dynamic-privileges) | Y | Y | Y | Y | N | N | +| [Security Enhanced Mode](/system-variables.md#tidb_enable_enhanced_security) | Y | Y | Y | Y | N | N | +| [Redacted Log Files](/log-redaction.md) | Y | Y | Y | Y | Y | N | ## Data import and export -| **Data import and export** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [Fast Importer (TiDB Lightning)](/tidb-lightning/tidb-lightning-overview.md) | Y | Y | Y | Y | Y | -| mydumper logical dumper | Deprecated | Deprecated | Deprecated | Deprecated | Deprecated | -| [Dumpling logical dumper](/dumpling-overview.md) | Y | Y | Y | Y | Y | -| [Transactional `LOAD DATA`](/sql-statements/sql-statement-load-data.md) | Y | Y | Y | Y | N [^3] | -| [Database migration toolkit (DM)](/migration-overview.md) | Y | Y | Y | Y | Y | -| [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) | Y | Y | Y | Y | Y | -| [Change data capture (CDC)](/ticdc/ticdc-overview.md) | Y | Y | Y | Y | Y | +| **Data import and export** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:| +| [Fast Importer (TiDB Lightning)](/tidb-lightning/tidb-lightning-overview.md) | Y | Y | Y | Y | Y | Y | +| mydumper logical dumper | Deprecated | Deprecated | Deprecated | Deprecated | Deprecated | Deprecated | +| [Dumpling logical dumper](/dumpling-overview.md) | Y | Y | Y | Y | Y | Y | +| [Transactional `LOAD DATA`](/sql-statements/sql-statement-load-data.md) | Y | Y | Y | Y | Y | N [^3] | +| [Database migration toolkit (DM)](/migration-overview.md) | Y | Y | Y | Y | Y | Y | +| [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) | Y | Y | Y | Y | Y | Y | +| [Change data capture (CDC)](/ticdc/ticdc-overview.md) | Y | Y | Y | Y | Y | Y | ## Management, observability, and tools -| **Management, observability, and tools** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | -|----------------------------------------------------------------------------------------------------------|:------------:|:------------:|:------------:|:------------:|:------------:| -| [TiDB Dashboard](/dashboard/dashboard-intro.md) | Y | Y | Y | Y | Y | -| [SQL diagnostics](/information-schema/information-schema-sql-diagnostics.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Information schema](/information-schema/information-schema.md) | Y | Y | Y | Y | Y | -| [Metrics schema](/metrics-schema.md) | Y | Y | Y | Y | Y | -| [Statements summary tables](/statement-summary-tables.md) | Y | Y | Y | Y | Y | -| [Slow query log](/identify-slow-queries.md) | Y | Y | Y | Y | Y | -| [TiUP deployment](/tiup/tiup-overview.md) | Y | Y | Y | Y | Y | -| Ansible deployment | N | N | N | N | Deprecated | -| [Kubernetes operator](https://docs.pingcap.com/tidb-in-kubernetes/) | Y | Y | Y | Y | Y | -| [Built-in physical backup](/br/backup-and-restore-use-cases.md) | Y | Y | Y | Y | Y | -| [Global Kill](/sql-statements/sql-statement-kill.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Lock View](/information-schema/information-schema-data-lock-waits.md) | Y | Y | Experimental | Experimental | Experimental | -| [`SHOW CONFIG`](/sql-statements/sql-statement-show-config.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [`SET CONFIG`](/dynamic-config.md) | Experimental | Experimental | Experimental | Experimental | Experimental | -| [Continuous Profiling](/dashboard/continuous-profiling.md) | Experimental | N | N | N | N | +| **Management, observability, and tools** | **5.4** | **5.3** | **5.2** | **5.1** | **5.0** | **4.0** | +| ------------------------------------------------------------ | ------------ | :----------: | :----------: | :----------: | :----------: | :----------: | +| [TiDB Dashboard](/dashboard/dashboard-intro.md) | Y | Y | Y | Y | Y | Y | +| [SQL diagnostics](/information-schema/information-schema-sql-diagnostics.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Information schema](/information-schema/information-schema.md) | Y | Y | Y | Y | Y | Y | +| [Metrics schema](/metrics-schema.md) | Y | Y | Y | Y | Y | Y | +| [Statements summary tables](/statement-summary-tables.md) | Y | Y | Y | Y | Y | Y | +| [Slow query log](/identify-slow-queries.md) | Y | Y | Y | Y | Y | Y | +| [TiUP deployment](/tiup/tiup-overview.md) | Y | Y | Y | Y | Y | Y | +| Ansible deployment | N | N | N | N | N | Deprecated | +| [Kubernetes operator](https://docs.pingcap.com/tidb-in-kubernetes/) | Y | Y | Y | Y | Y | Y | +| [Built-in physical backup](/br/backup-and-restore-use-cases.md) | Y | Y | Y | Y | Y | Y | +| [Global Kill](/sql-statements/sql-statement-kill.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Lock View](/information-schema/information-schema-data-lock-waits.md) | Y | Y | Y | Experimental | Experimental | Experimental | +| [`SHOW CONFIG`](/sql-statements/sql-statement-show-config.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [`SET CONFIG`](/dynamic-config.md) | Experimental | Experimental | Experimental | Experimental | Experimental | Experimental | +| [Continuous Profiling](/dashboard/continuous-profiling.md) | Experimental | Experimental | N | N | N | N | +| [Top SQL](/dashboard/top-sql.md) | Experimental | N | N | N | N | N | [^1]: TiDB incorrectly treats latin1 as a subset of utf8. See [TiDB #18955](https://github.com/pingcap/tidb/issues/18955) for more details. diff --git a/experimental-features.md b/experimental-features.md index cb457c7a7fefe..4234837051ad4 100644 --- a/experimental-features.md +++ b/experimental-features.md @@ -7,6 +7,13 @@ summary: Learn the experimental features of TiDB. This document introduces the experimental features of TiDB in different versions. It is **NOT** recommended to use these features in the production environment. +## Performance + ++ [Automatically scale TiFlash thread pool](/tiflash/tiflash-configuration.md). (Introduced in v5.4) ++ [Support synchronously loading statistics](/statistics.md#load-statistics). (Introduced in v5.4) ++ [Raft Engine](/tikv-configuration-file.md#raft-engine) (Introduced in v5.4) ++ [Support collecting statistics for `PREDICATE COLUMNS`](/statistics.md#collect-statistics-for-some-columns) (Introduced in v5.4) + ## Stability + TiFlash limits the use of I/O resources by compressing or sorting data, mitigating the contention for I/O resources between background tasks and front-end data reading and writing (Introduced in v5.0) @@ -46,10 +53,6 @@ This document introduces the experimental features of TiDB in different versions + [Titan Level Merge](/storage-engine/titan-configuration.md#level-merge-experimental) (Introduced in v4.0) + TiFlash supports distributing the new data of the storage engine on multiple hard drives to share the I/O pressure. (Introduced in v4.0) -## Data migration - -+ [DM OpenAPI](/dm/dm-open-api.md) (Introduced in v5.3) - ## Backup and restoration + [Back up Raw KV](/br/use-br-command-line-tool.md#back-up-raw-kv-experimental-feature) (Introduced in v3.1) @@ -63,4 +66,5 @@ This document introduces the experimental features of TiDB in different versions + [SQL diagnostics](/information-schema/information-schema-sql-diagnostics.md) (Introduced in v4.0) + [Cluster diagnostics](/dashboard/dashboard-diagnostics-access.md) (Introduced in v4.0) + [Continuous profiling](/dashboard/continuous-profiling.md) (Introduced in v5.3) -+ [Online Unsafe Recovery](/online-unsafe-recovery.md) (Introduced in v5.3) \ No newline at end of file ++ [Online Unsafe Recovery](/online-unsafe-recovery.md) (Introduced in v5.3) ++ [Top SQL](/dashboard/top-sql.md) (Introduced in v5.4) From 2f4a163e072b91699c933945d93c1dc08dc0c93e Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 21 Jan 2022 12:07:36 +0800 Subject: [PATCH 05/27] update according to google docs --- releases/release-5.4.0.md | 186 +++++++++++++++++++------------------- 1 file changed, 93 insertions(+), 93 deletions(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index ea7f1e8ca5370..6df9bb8fee885 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -17,8 +17,8 @@ In v5.4, the key new features or improvements are as follows: - Support using Raft Engine as the log storage engine of TiKV (experimental) - Optimize the impact of backup on the cluster - Support using Azure Blob storage as the target storage for backup -- Continuouly improve the stability and performance of TiFlash and the MPP engine -- Add a switch in TiDB Lightning to determine whether to allow importing an existing table +- Continuously improve the stability and performance of TiFlash and the MPP engine +- Add a switch in TiDB Lightning to determine whether to allow importing to an existing table with data - Optimize the Continuous Profiling feature (experimental) - TiSpark supports user identification and authentication @@ -26,67 +26,67 @@ In v5.4, the key new features or improvements are as follows: > **Note:** > -> When upgrading from an earlier TiDB version to v5.4.0, if you want to know the compatibility change notes of all intermediate versions, you can check the [Release Notes](/releases/release-notes.md) of the corresponding version. +> When upgrading from an earlier TiDB version to v5.3.0, if you want to know the compatibility change notes of all intermediate versions, you can check the [Release Notes](/releases/release-notes.md) of the corresponding version. ### System variables | Variable name | Change type | Description | | :---------- | :----------- | :----------- | -| [`tidb_backoff_lock_fast`](/system-variables.md#tidb_backoff_lock_fast) | Modified | The default value is changed from `100` to `10`. | -| [`tidb_enable_column_tracking`](/system-variables.md#tidb_enable_column_tracking-new-in-v50) | Newly added | Controls whether allow TiDB to collect `PREDICATE COLUMNS`. The default value is `OFF`. | -| [`tidb_enable_index_merge`](/system-variables.md#tidb_enable_index_merge-new-in-v40) | Modified | The default value is changed from `OFF` to `ON`.
| -| [`tidb_enable_paging`](/system-variables.md#tidb_enable_paging-new-in-v540) | Newly added | This variable controls whether to use the method of paging to send coprocessor requests in `IndexLookUp` operator. The default value is `OFF`.
For read queries that use `IndexLookup` and `Limit` and that `Limit` cannot be pushed down to `IndexScan`, there might be high latency for the read queries and high CPU usage for TiKV's `unified read pool`. In such cases, because the `Limit` operator only requires a small set of data, if you set `tidb_enable_paging` to `ON`, TiDB processes less data, which reduces query latency and resource consumption. | +| [`tidb_enable_column_tracking`](/system-variables.md#tidb_enable_column_tracking-new-in-v540) | Newly added | Controls whether to allow TiDB to collect `PREDICATE COLUMNS`. The default value is `OFF`. | +| [`tidb_enable_paging`](/system-variables.md#tidb_enable_paging-new-in-v540) | Newly added | Controls whether to use the method of paging to send coprocessor requests in `IndexLookUp` operator. The default value is `OFF`.
For read queries that use `IndexLookup` and `Limit` and that `Limit` cannot be pushed down to `IndexScan`, there might be high latency for the read queries and high CPU usage for TiKV's `unified read pool`. In such cases, because the `Limit` operator only requires a small set of data, if you set `tidb_enable_paging` to `ON`, TiDB processes less data, which reduces query latency and resource consumption. | | [`tidb_enable_top_sql`](/system-variables.md#tidb_enable_top_sql-new-in-v540) | Newly added | Controls whether to enable the Top SQL feature. The default value is `OFF`. | | [`tidb_persist_analyze_options`](/system-variables.md#tidb_persist_analyze_options-new-in-v540) | Newly added | Controls whether to enable the [ANALYZE configuration persistence](/statistics.md#persist-analyze-configurations) feature. The default value is `OFF`. | | [`tidb_read_staleness`](/system-variables.md#tidb_read_staleness-new-in-v540) | Newly added | Controls the range of historical data that can be read in the current session. The default value is `0`.| -| [`tidb_regard_null_as_point `](/system-variables.md#tidb_regard_null_as_point-new-in-v540) | Newly added | Controls whether the optimizer can treat `null` as a point value and uses it as a prefix condition to access the index 用于控制优化器是否可以把 null 值当做点值并作为前缀条件来访问索引 | -| [`tidb_stats_load_sync_wait`](/system-variables.md#tidb_stats_load_sync_wait-new-in-v540) | Newly added | This variable controls whether to enable the synchronously loading statistics feature. (The default value `0` means that the feature is disabled and that the statistics is asynchronously loaded.) When the feature is enabled, this variable controls the maximum time that SQL optimization can wait for synchronously loading statistics before timeout. | -| [`tidb_stats_load_pseudo_timeout`](/system-variables.md#tidb_stats_load_pseudo_timeout-new-in-v540) | Newly added | This variable controls when synchronously loading statistics reaches timeout, whether SQL fails (`OFF`) or falls back to using pseudo statistics. The default value is `ON`. | -| [`tidb_store_limit`](/system-variables.md#tidb_store_limit-new-i-v304-and-v40) | Modified | Before v5.4.0, this variabe can be configured at instance level and globally. Starting from v5.4.0, this variable only supports global configuration. | +| [`tidb_regard_null_as_point `](/system-variables.md#tidb_regard_null_as_point-new-in-v540) | Newly added | Controls whether the optimizer can treat `null` as a point value and uses it as a prefix condition to access the index. | +| [`tidb_stats_load_sync_wait`](/system-variables.md#tidb_stats_load_sync_wait-new-in-v540) | Newly added | Controls whether to enable the synchronously loading statistics feature. The default value `0` means that the feature is disabled and that the statistics is asynchronously loaded. When the feature is enabled, this variable controls the maximum time that SQL optimization can wait for synchronously loading statistics before timeout. | +| [`tidb_stats_load_pseudo_timeout`](/system-variables.md#tidb_stats_load_pseudo_timeout-new-in-v540) | Newly added | Controls when synchronously loading statistics reaches timeout, whether SQL fails (`OFF`) or falls back to using pseudo statistics (`ON`). The default value is `ON`. | +| [`tidb_backoff_lock_fast`](/system-variables.md#tidb_backoff_lock_fast) | Modified | The default value is changed from `100` to `10`. | +| [`tidb_enable_index_merge`](/system-variables.md#tidb_enable_index_merge-new-in-v40) | Modified | The default value is changed from `OFF` to `ON`.
| +| [`tidb_store_limit`](/system-variables.md#tidb_store_limit-new-i-v304-and-v40) | Modified | Before v5.4.0, this variable can be configured at instance level and globally. Starting from v5.4.0, this variable only supports global configuration. | ### Configuration file parameters | Configuration file | Configuration | Change type | Description | | :---------- | :----------- | :----------- | :----------- | -| TiDB | [`stats-load-concurrency`](/tidb-configuration-file.md#stats-load-concurrency-new-in-v540) | Newly added | Controls the maximum number of columns that the TiDB synchronously loading statistics feature can process concurrently. The default value is `5` | +| TiDB | [`stats-load-concurrency`](/tidb-configuration-file.md#stats-load-concurrency-new-in-v540) | Newly added | Controls the maximum number of columns that the TiDB synchronously loading statistics feature can process concurrently. The default value is `5`. | | TiDB | [`stats-load-queue-size`](/tidb-configuration-file.md#stats-load-queue-size-new-in-v540) | Newly added | Controls the maximum number of column requests that the TiDB synchronously loading statistics feature can cache. The default value is`1000`. | | TiKV | [`snap-generator-pool-size`](/tidb-configuration-file.md#snap-generator-pool-size) | Newly added | The size of `snap-generator` thread pool. The default value is `2`. | -| TiKV | `log.file.max-size`、`log.file.max-days`、`log.file.max-backups` | Newly added | For details, see [TiKV Configuration File - log.file](/tikv-configuration-file.md#logfile-new-in-v540). | -| TiKV | `raft-engine` | Newly added | Includes `enable`, `dir`, `batch-compression-threshold`, `bytes-per-sync`, `target-file-size`, `purge-threshold`, `recovery-mode`, `recovery-read-block-size`, `recovery-read-block-size`, `recovery-threads`. For details, see [TiKV Configuration File - Raft-engine](/tikv-configuration-file.md#raft-engine).| -| TiKV | [`backup.enable-auto-tune`](/tikv-configuration-file.md#enable-auto-tune-new-in-v540) | Newly added | In v5.3.0, the default value is `false`. Since v5.4.0, the default value is changed to `true`. This parameter controls whether to limit the resources used by backup tasks to reduce the impact on the cluster when the cluster resource utilization is high. In the default configuration, the speed of backup task might slow down. | -| TiKV | `log-level`, `log-format`, `log-file`, `log-rotation-size` | Modified | The names of TiKV log parameters are replaced with the names that are same as TiDB log parameters, which are `log.level`, `log.format`, `log.file.filename`, `log.enable-timestamp`. If you only set the old parameters, and their values are set to non-default values, the old parameters remain compatible with the new parameters. If both old and new parameters are set, the new parameters take effect. For details, see [TiKV Configuration File - log](/tikv-configuration-file.md#log-new-in-v540). | -| TiKV | `log-rotation-timespan` | Deleted | The timespan between log rotations. When this timespan passes, log files are rotated, that is, a timestamp is appended to the file name of the current log file, and a new file is created. | +| TiKV | `log.file.max-size`, `log.file.max-days`, `log.file.max-backups` | Newly added | For details, see [TiKV Configuration File - `log.file`](/tikv-configuration-file.md#logfile-new-in-v540). | +| TiKV | `raft-engine` | Newly added | Includes `enable`, `dir`, `batch-compression-threshold`, `bytes-per-sync`, `target-file-size`, `purge-threshold`, `recovery-mode`, `recovery-read-block-size`, `recovery-read-block-size`, and `recovery-threads`. For details, see [TiKV Configuration File - `raft-engine`](/tikv-configuration-file.md#raft-engine).| +| TiKV | [`backup.enable-auto-tune`](/tikv-configuration-file.md#enable-auto-tune-new-in-v540) | Newly added | In v5.3.0, the default value is `false`. Since v5.4.0, the default value is changed to `true`. This parameter controls whether to limit the resources used by backup tasks to reduce the impact on the cluster when the cluster resource utilization is high. In the default configuration, the speed of backup tasks might slow down. | +| TiKV | `log-level`, `log-format`, `log-file`, `log-rotation-size` | Modified | The names of TiKV log parameters are replaced with the names that are consistent with TiDB log parameters, which are `log.level`, `log.format`, `log.file.filename`, and `log.enable-timestamp`. If you only set the old parameters, and their values are set to non-default values, the old parameters remain compatible with the new parameters. If both old and new parameters are set, the new parameters take effect. For details, see [TiKV Configuration File - log](/tikv-configuration-file.md#log-new-in-v540). | +| TiKV | `log-rotation-timespan` | Deleted | The timespan between log rotations. After this timespan passes, a log file is rotated, which means a timestamp is appended to the file name of the current log file, and a new log file is created. | | TiKV | `allow-remove-leader` | Deleted | Determines whether to allow deleting the main switch. | -| TiKV | `raft-msg-flush-interval` | Deleted | Determines the interval at which Raft messages are sent in batches. The Raft messages in batches are sent at every interval specified by this configuration item. | -| PD | [`log.level`](/pd-configuration-file.md#level) | Modified | The default value is changed from "INFO" to "info", guaranteed to be case in-sensitive. | +| TiKV | `raft-msg-flush-interval` | Deleted | Determines the interval at which Raft messages are sent in batches. The Raft messages are sent in batches at every interval specified by this configuration item. | +| PD | [`log.level`](/pd-configuration-file.md#level) | Modified | The default value is changed from "INFO" to "info", guaranteed to be case-insensitive. | | TiFlash | [`profile.default.enable_elastic_threadpool`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Newly added | Determines whether to enable or disable the elastic thread pool function. Enabling this configuration item can significantly improve TiFlash CPU utilization in high concurrency scenarios. The default value is `false`. | | TiFlash | [`storage.format_version`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Newly added | Specifies the version of DTFile. The default value is `2`, under which hashes are embedded in the data file. You can also set the value to `3`. When it is `3`, the data file contains metadata and token data checksum, and supports multiple hash algorithms. | | TiFlash | [`logger.count`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Modified | The default value is changed to `10`. | | TiFlash | [`status.metrics_port`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Modified | The default value is changed to `8234`. | | TiFlash | [`raftstore.apply-pool-size`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) | Newly added | The allowable number of threads in the pool that flushes Raft data to storage. The default value is `4`. | | TiFlash | [`raftstore.store-pool-size`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) | Newly added | The allowable number of threads that process Raft, which is the size of the Raftstore thread pool. The default value is `4`. | -| TiDB Data Migration (DM) | [`collation_compatible`](/dm/task-configuration-file-full.md##task-configuration-file-template-advanced) | Newly added | The mode to sync the default collation in `CREATE` SQL statements. The supported values are "loose" (by default) or "strict". | +| TiDB Data Migration (DM) | [`collation_compatible`](/dm/task-configuration-file-full.md#task-configuration-file-template-advanced) | Newly added | The mode to sync the default collation in `CREATE` SQL statements. The value options are "loose" (by default) and "strict". | | TiCDC | `max-message-bytes` | Modified| Change the default value of `max-message-bytes` in Kafka sink to `104857601` (10MB) | | TiCDC | `partition-num` | Modified | Change the default value of `partition-num` in Kafka Sink from `4` to `3`. It makes TiCDC send messages to Kafaka partitions more evenly. | | TiDB Lightning | `meta-schema-name` | Modified | Specifies the schema name for the metadata in the target TiDB. From v5.4.0, this schema is created only if you have enabled [parallel import](/tidb-lightning/tidb-lightning-distributed-import) (the corresponding parameter is `tikv-importer.incremental-import = true`). | -| TiDB Lightning | `task-info-schema-name` | Newly added | Specifies the name of the database where duplicated data is stored when Lightning detects conflicts. By default, the value is "lightning_task_info". Specify this parameter only if you have enabled the "duplicate-resolution" feature. | -| TiDB Lightning | `incremental-import` | Newly added | Whether to allow importing data to tables where data already exists. The default value is `false`. | - +| TiDB Lightning | `task-info-schema-name` | Newly added | Specifies the name of the database where duplicated data is stored when TiDB Lightning detects conflicts. By default, the value is "lightning_task_info". Specify this parameter only if you have enabled the "duplicate-resolution" feature. | +| TiDB Lightning | `incremental-import` | Newly added | Determines whether to allow importing data to tables where data already exists. The default value is `false`. | ### Others -- TiDB Dashboard, by default, does not permit using `root` + empty password for login. - Since v5.4.0, it is recommended to use `start --initial` when you use TiUP to start a TiDB cluster. Then, a random password is generated for the `root` account to log into TiDB Dashboard. +- TiDB Dashboard, by default, does not permit using `root` and empty password for login. -- An interface is added between TiDB and PD. When using the `information_schema.TIDB_HOT_REGIONS_HISTORY` system table, TiDB needs to use the PD in a matching version. -- TiDB Server, PD Server, and TiKV Server start using a unified naming method for the log-related parameters to manage log names, output formats, the rules of rotation and expiration. For details, see [TiKV configuration file - log](/tikv-configuration-file.md#log-new-in-v540). -- Since v5.4.0, if you create a SQL binding for an execution plan that has been cached via Plan Cache, the binding invalidates the plan already cached for the corresponding query. Before v5.4.0, the new binding does not affect any execution plans that have been already cached. + Since v5.4.0, it is recommended to use `start --initial` when you use TiUP to start a TiDB cluster. Then, a random password is generated for the `root` account to log into TiDB Dashboard. + +- An interface is added between TiDB and PD. When you use the `information_schema.TIDB_HOT_REGIONS_HISTORY` system table, TiDB needs to use PD in the corresponding version. +- TiDB Server, PD Server, and TiKV Server start using a unified naming method for the log-related parameters to manage log names, output formats, and the rules for rotation and expiration. For details, see [TiKV configuration file - log](/tikv-configuration-file.md#log-new-in-v540). +- Since v5.4.0, if you create a SQL binding for an execution plan that has been cached via Plan Cache, the binding invalidates the plan already cached for the corresponding query. The new binding does not affect execution plans cached before v5.4.0. ## New features ### SQL -- **TiDB supports GBK since v5.4.0** +- **TiDB supports the GBK character set since v5.4.0** Before v5.4.0, TiDB supports `ascii`, `binary`, `latin1`, `utf8`, and `utf8mb4` character sets. @@ -106,26 +106,28 @@ In v5.4, the key new features or improvements are as follows: ### Performance -- **Continue improving the stability and performance of the column-wise storage engine and the MPP computing engine for TiFlash** +- **Continue improving the stability and performance of the columnar storage engine TiFlash and the computing engine MPP** + + - Support pusing down more functions to the MPP engine: - - Support more functions to be pushed down to the MPP engine: - String functions: `LPAD()`, `RPAD()`, `STRCMP()` - Date functions: `ADDDATE()`, `DATE_ADD()`, `DATE_SUB()`, `SUBDATE()`, `QUARTER()` - - Introduce the elastic thread pool function to improve resource utilization (experimental feature) - - Accelerate row-column data conversion when replicating data from TiKV. This improves overall replication performance by 50%. - - Improve TiFlash performance and stability by adjusting the default value of some configuration items, yielding up to 20% performance improvement in simple queries per table. + + - Introduce the elastic thread pool feature to improve resource utilization (experimental) + - Improve the efficiency of converting data from row-based storage format to column-based storage format when replicating data from TiKV, which brings 50% improvement in the overall performance of data replication + - Improve TiFlash performance and stability by tuning the default values of some configuration items. In an HTAP hybrid load, the performance of simple queries on a single table improves up to 20%. User documents: [Supported push-down calculations](/tiflash/use-tiflash.md#supported-push-down-calculations), [Configure the tiflash.toml file](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) -- **Read expired data within a specified time range through a session variable** +- **Read historical data within a specified time range through a session variable** - TiDB is a multi-replica distributed database that uses the Raft protocol. In the high concurrency and high throughput business scenarios, TiDB can make read performance scalable through follower nodes and build a read-write separation architecture. + TiDB is a multi-replica distributed database based on the Raft consensus algorithm. In face of high-concurrency and high-throughput application scenarios, TiDB can scale out its read performance through follower replicas, separating read and write requests. - For different business scenarios, the follower provides two read modes: strongly consistent read and weakly consistent expired read. The strongly consistent read mode is suitable for use in business scenarios with strict data real-time requirements. However, when this mode is used, especially in the geo-distributed deployment, there will be latency issue due to the data replication latency and the reduced throughput of the leader and the follower. + For different application scenarios, TiDB provides two modes of follower read: strongly consistent read and weakly consistent history read. The strongly consistent read mode is suitable for application scenarios that require real-time data. However, in this mode, because of the data replication latency between leaders and followers and the reduced throughput, the read request might have high latency, especially for geo-distributed deployments. - For the business scenarios that do not require high real-time performance, the expired read mode can be used. This mode can reduce latency and improve throughput. TiDB currently supports the expired read by displaying read-only transactions or SQL statements. Both methods support reading the historical data of a specific point in time or within a specified time range. For details, please refer to [Read Historical Data through session variable `tidb_snapshot`](/read-historical-data.md). + For the application scenarios that have less strict requirements on real-time data, the history read mode is recommended. This mode can reduce latency and improve throughput. TiDB currently supports reading historical data by the following methods: Use SQL statements to read data from a time point in the past, or start a read-only transaction based on a time point in the past. Both methods support reading the historical data of a specific point in time or within a specified time range. For details, refer to [Read Historical Data Using the `AS OF TIMESTAMP` Clause](/as-of-timestamp.md). - Since v5.4.0, TiDB supports reading historical data within a specified time range a through session variable, which further improves usability, reduces the difficulty of development, and meets the business requirements of low latency and high throughput of data access in quasi-real-time scenarios. You can set the variable as following example: + Since v5.4.0, TiDB improves the usability of the history read mode by supporting reading historical data within a specified time range through a session variable. This mode serves low-latency, high-throughput read requests in quasi-real-time scenarios. You can set the variable as follows: ```sql set @@tidb_replica_read=leader_and_follower @@ -134,23 +136,23 @@ In v5.4, the key new features or improvements are as follows: By this setting, TiDB can select the nearest leader or follower node and read the latest historical data within 5 seconds. - [User document](/read-historical-data.md) + [User document](/tidb-read-staleness.md) - **GA for Index Merge** - _Index Merge_ is introduced in TiDB v4.0 as an experimental feature to access tables. This method greatly accelerates condition filtering when a query requires scanning of multiple columns of data. Take the following query as an example. In the `WHERE` statement, the filtering conditions connected by `OR` have their respective indexes in columns _key1_ and _key2_. Based on the indexes in the two columns, the Index Merge feature filters and merges the query results, and returns the merged result. + Index Merge is introduced in TiDB v4.0 as an experimental feature for SQL optimization. This method greatly accelerates condition filtering when a query requires scanning of multiple columns of data. Take the following query as an example. In the `WHERE` statement, if the filtering conditions connected by `OR` have their respective indexes on columns _key1_ and _key2_, the Index Merge feature filters the respective indexes at the same time, merges the query results, and returns the merged result. ```sql SELECT * FROM table WHERE key1 <= 100 OR key2 = 200; ``` - Before TiDB v4.0, a query in a table uses only one not multiple indexes for filtering. If you want to query multiple columns of data, you can enable _Inex Merge_ to get the exact query result in short time by using the indexes in individual columns. _Inex Merge_ avoids unnecessary full table scans and does not require establishing large number of composite indexes. + Before TiDB v4.0, a query on a table supports using only one index for filtering at one time. If you want to query multiple columns of data, you can enable Index Merge to get the exact query result in a short time by using the indexes in individual columns. Index Merge avoids unnecessary full table scans and does not require establishing a large number of composite indexes. - In v5.4.0, _Inex Merge_ is a GA feature. However, you still need to pay attention to the following restrictions: + In v5.4.0, Inex Merge becomes GA. However, you still need to pay attention to the following restrictions: - - _Inex Merge_ supports only disjunctive normal form (X1 ⋁ X2 ⋁ …Xn). That is, to use this feature, filtering conditions in a `WHERE` clause should be connected by `OR`. + - Index Merge supports only disjunctive normal form (X1 ⋁ X2 ⋁ …Xn). That is, this feature only works when filtering conditions in a `WHERE` clause are connected by `OR`. - - This feature is enabled by default for TiDB clusters of v5.4.0 or later. For v5.4.0 or later TiDB clusters upgraded from versions earlier than v5.4.0, this feature inherits the setting before the upgrade and you can change the setting as required (in TiDB clusters earlier than v4.0, this feature does not exist and is disabled by default). + - For newly deployed TiDB clusters of v5.4.0 or later, this feature is enabled by default. For v5.4.0 or later TiDB clusters upgraded from earlier versions, this feature inherits the setting before the upgrade and you can change the setting as required (for TiDB clusters earlier than v4.0, this feature does not exist and is disabled by default). [User document](/explain-index-merge.md) @@ -160,7 +162,7 @@ In v5.4, the key new features or improvements are as follows: Raft Engine is still an experimental feature and is disabled by default. Note that the data format of Raft Engine in v5.4.0 is not compatible with previous versions. Before upgrading or downgrading the cluster, you need to make sure that Raft Engine on all TiKV nodes is disabled. It is recommended to use Raft Engine only in v5.4.0 or a later version. -- **Support collecting statistics for `PREDICATE COLUMNS` (experimental)** +- **Support collecting statistics on `PREDICATE COLUMNS` (experimental)** In most cases, when executing SQL statements, the optimizer only uses statistics of some columns (such as columns in the `WHERE`, `JOIN`, `ORDER BY`, and `GROUP BY` statements). These used columns are called `PREDICATE COLUMNS`. @@ -172,9 +174,9 @@ In v5.4, the key new features or improvements are as follows: - **Support synchronously loading statistics (experimental)** - Since v5.4.0, TiDB introduces the synchronously loading statistics feature. The feature is disabled by default. After enabling the feature, TiDB can synchronously load statistics with a large space consumption (such as histogram, TopN, and Count-Min Sketch statistics) into memory when you execute SQL statements, which improves the completeness of statistics for SQL optimization. + Since v5.4.0, TiDB introduces the synchronously loading statistics feature. This feature is disabled by default. After enabling the feature, TiDB can synchronously load large-sized statistics (such as histograms, TopN, and Count-Min Sketch statistics) into memory when you execute SQL statements, which improves the completeness of statistics for SQL optimization. - [[User document](/statistics.md#load-statistics) + [User document](/statistics.md#load-statistics) ### Stability @@ -184,7 +186,7 @@ In v5.4, the key new features or improvements are as follows: Since v5.4.0, TiDB supports persisting some `ANALYZE` configurations. With this feature, the existing configurations can be easily reused for future statistics collection. - The `ANALYZE` configuration persistence feature is enabled by default (the system variable `tidb_analyze_version` is `2` and `tidb_persist_analyze_options` is `ON` by default). You can use this feature to record the persistence configurations specified in the `ANALYZE` statement when executing the statement manually. Once recorded, the next time TiDB automatically updates statistics or you manually collect statistics without specifying these configuration, TiDB will collect statistics according to the recorded configurations. + The `ANALYZE` configuration persistence feature is enabled by default (the system variable `tidb_analyze_version` is `2` and `tidb_persist_analyze_options` is `ON` by default). You can use this feature to record the persistence configurations specified in the `ANALYZE` statement when executing the statement manually. Once recorded, the next time TiDB automatically updates statistics or you manually collect statistics without specifying these configurations, TiDB will collect statistics according to the recorded configurations. [User document](/statistics.md#persist-analyze-configurations) @@ -192,13 +194,13 @@ In v5.4, the key new features or improvements are as follows: - **Reduce the impact of backup tasks on the cluster** - Backup & Restore (BR) introduces the auto-tune feature (enabled by default). This feature monitors the cluster resource usage and adjusts the number of threads used by the backup tasks to reduce the impact of backup tasks on the cluster. In some cases, if you increase the cluster resources for backup and enable the auto-tune feature, it is possible to limit the impact of backup tasks on the cluster to 10% or less. + Backup & Restore (BR) introduces the auto-tune feature (enabled by default). This feature can reduce the impact of backup tasks on the cluster by monitoring the cluster resource usage and adjusting the number of threads used by the backup tasks. In some cases, if you increase the cluster resource usage for backup and enable the auto-tune feature, it is possible to limit the impact of backup tasks on the cluster to 10% or less. - [User document](/br/br-auto-tune.md) + [User document](/br/br-auto-tune.md) - **Support Azure Blob Storage as a target storage for backup** - Backup & Restore (BR) supports Azure Blob Storage as a remote target storage for backup. If you deploy TiDB in Azure Cloud, you can back up the cluster data to the Azure Blob Storage service. + Backup & Restore (BR) supports Azure Blob Storage as a remote target storage for backup tasks. If you deploy TiDB in Azure Cloud, you can back up the cluster data to the Azure Blob Storage service. [User document](/br/backup-and-restore-azblob.md) @@ -206,30 +208,33 @@ In v5.4, the key new features or improvements are as follows: - **TiDB Lightning introduces a new feature to determine whether to allow importing data to tables with data** - TiDB Lightning introduces a new feature `incremental-import`. It determines whether to allow importing data to tables with data. The default value is `false`. When using parallel import mode, you must set it to `true`. + TiDB Lightning introduces a configuration item`incremental-import`. It determines whether to allow importing data to tables with data. The default value is `false`. When using the parallel import mode, you must set the configuration to `true`. [User document](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) - **TiDB Lightning introduces the schema name that stores the meta information for parallel import** - TiDB Lightning introduces `meta-schema-name`. In parallel import mode, this parameter specifies the schema name that stores the meta information for each TiDB Lightning instance in the target cluster. By default, the value is "lightning_metadata". The value set for this parameter must be the same for each TiDB Lightning instance that participates in the same parallel import; otherwise, the correctness of the imported data can not be ensured. + TiDB Lightning introduces the `meta-schema-name` configuration item. In parallel import mode, this parameter specifies the schema name that stores the meta information for each TiDB Lightning instance in the target cluster. By default, the value is "lightning_metadata". The value set for this parameter must be the same for each TiDB Lightning instance that participates in the same parallel import; otherwise, the correctness of the imported data can not be ensured. - [User document](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) + [User document](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) - **TiDB Lightning introduces duplicate resolution** - In Local-backend mode, TiDB Lightning outputs duplicated data before the data import is completed, and then removes that duplicated data from the database. You can resolve the duplicated data after the import is completed and select suitable data to insert according to business rules. It is recommended to clean upstream data sources based on duplicated data to avoid data inconsistency caused by duplicated data encountered in the subsequent incremental data migration phase. + In Local-backend mode, TiDB Lightning outputs duplicated data before the data import is completed, and then removes that duplicated data from the database. You can resolve the duplicated data after the import is completed and select suitable data to insert according to application rules. It is recommended to clean upstream data sources based on duplicated data to avoid data inconsistency caused by duplicated data encountered in the subsequent incremental data migration phase. [User document](tidb-lightning/tidb-lightning-error-resolution.md) - **Optimize the usage of relay log in TiDB Data Migration (DM)** - Recover the `enable-relay` switch in the `source` configuration. + - Support dynamically enabling or disabling relay log in the `start-relay` or `stop-relay` command. + - Bind the status of relay log to `source`. `source` keeps its original status of being enabled or disabled after it is migrated to any DM-worker. - - Move the storage path of relay log to the DM-worker configuration file. - [User document](/dm/relay-log.md) + - Move the storage path of relay log to the DM-worker configuration file. + + [User document](/dm/relay-log.md) - **Optimize the processing of [collation](/character-set-and-collation.md) in DM** @@ -238,11 +243,11 @@ In v5.4, the key new features or improvements are as follows: - If your application does not have strict requirements on collation, and the collation of query results can be different between the upstream and downstream, you can use the default `loose` mode to avoid reporting errors. - If your application has strict requirements on collation, and the collation must be consistent between the upstream and downstream, you can use the `strict` mode. However, if the downstream does not support the upstream's default collation, the data replication might report errors. - [User document](/dm/task-configuration-file-full.md#task-configuration-file-template-advanced) + [User document](/dm/task-configuration-file-full.md#task-configuration-file-template-advanced) -- **Optimize `transfer source` in DM to support smooth execution of replication tasks** +- **Optimize `transfer source` in DM to support running replication tasks smoothly** - When the DM-worker nodes have unbalanced load, the `transfer source` command can be used to manually transfer the configuration of a `source` to another load. After the optimization, the `transfer source` command simplifies the manual operation. You can smoothly transfer the source instead of pausing all related tasks; DM completes other operations internally. + When the DM-worker nodes have an unbalanced load, the `transfer source` command can be used to manually transfer the configuration of a `source` to another load. After the optimization, the `transfer source` command simplifies the manual operation. You can smoothly transfer the source instead of pausing all related tasks, because DM completes other operations internally. - **DM OpenAPI becomes generally available (GA)** @@ -254,7 +259,7 @@ In v5.4, the key new features or improvements are as follows: - **Top SQL (experimental feature)** - A new experimental feature, Top SQL (turned off by default), is introduced to help you easily find queries that consume more resources. + A new experimental feature, Top SQL (disabled by default), is introduced to help you easily find source-consuming queries. [User document](/dashboard/top-sql.md) @@ -262,19 +267,20 @@ In v5.4, the key new features or improvements are as follows: - **Optimize the impact of TiCDC on clusters** - Significantly reduces the performance impact on TiDB clusters when you use TiCDC. In the test environment, the performance impact of TiCDC on TiDB can be reduced to less than 5%. + Significantly reduces the impact on the performance of TiDB clusters when you use TiCDC. In the test environment, the performance impact of TiCDC on TiDB can be reduced to less than 5%. ### Deployment and maintenance -- **Enhance Continuous Profiling (experimental feature)** +- **Enhance Continuous Profiling (experimental)** - More components supported: Besides TiDB, PD, and TiKV, TiDB v5.4.0 also supports CPU profiling of TiFlash. - - More forms of profiling display: Supports display of CPU profiling and Goroutine results on flame charts. - - More deployment environments supported: Continuous Profiling is applicable to clusters deployed using TiDB Operator. + - More forms of profiling display: Supports showing CPU profiling and Goroutine results on flame charts. + + - More deployment environments supported: Continuous Profiling can also be used for clusters deployed using TiDB Operator. - Continuous Profiling is disabled by default and can be enabled on TiDB Dashboard. + Continuous Profiling is disabled by default and can be enabled on TiDB Dashboard. - Continuous Profiling is available to clusters deployed or upgraded using TiUP of v1.9.0 or later or TiDB Operator of v1.3.0 or later. + Continuous Profiling is applicable to clusters deployed or upgraded using TiUP of v1.9.0 or later or TiDB Operator of v1.3.0 or later. [User document](/dashboard/continuous-profiling.md) @@ -282,7 +288,7 @@ In v5.4, the key new features or improvements are as follows: + TiDB - - Add a new system variable `tidb_enable_paging. For read requests in which `IndexLookUp` and `Limit` are used and `Limit` cannot be pushed down to `IndexScan`, setting this variable to `ON` can significantly reduce the latency and the resource consumption [#30578](https://github.com/pingcap/tidb/issues/30578) + - Add a new system variable `tidb_enable_paging` to determine whether to use paging to send Coprocessor requests. Enabling this feature can reduce the amount of data to process and to reduce latency and resource consumption [#30578](https://github.com/pingcap/tidb/issues/30578) - Support the `ADMIN {SESSION | INSTANCE | GLOBAL} PLAN_CACHE` syntax to clear the cached query plan [#30370](https://github.com/pingcap/tidb/pull/30370) + TiKV @@ -295,7 +301,7 @@ In v5.4, the key new features or improvements are as follows: - Coprocessor supports pushing down substring to TiKV [#11495](https://github.com/tikv/tikv/issues/11495) - Improve the scan performance by skip reading locks in the Read Committed isolation level [#11485](https://github.com/tikv/tikv/issues/11485) - Reduce the default thread pool size used by backup operations and limit the use of thread pool when the stress is high [#11000](https://github.com/tikv/tikv/issues/11000) - - Support dynamically adjusting the sizes of Apply thread pool and Store thread pool [#11159](https://github.com/tikv/tikv/issues/11159) + - Support dynamically adjusting the sizes of the Apply thread pool and the Store thread pool [#11159](https://github.com/tikv/tikv/issues/11159) - Support configuring the size of the `snap-generator` thread pool [#11247](https://github.com/tikv/tikv/issues/11247) - Optimize the issue of global lock race that occurs when there are many files with frequent reads and writes [#250](https://github.com/tikv/rocksdb/pull/250) @@ -308,13 +314,13 @@ In v5.4, the key new features or improvements are as follows: + TiFlash - Optimize the communication of local operators - - Increase the non-temporary thread count of gRPC to avoid the frequent creation or destroy of threads + - Increase the non-temporary thread count of gRPC to avoid the frequent creation or destruction of threads + Tools + Backup & Restore (BR) - - Add a validity check for the key when BR performs encrypted backup [#29794](https://github.com/pingcap/tidb/issues/29794) + - Add a validity check for the key when BR performs encrypted backup [#29794](https://github.com/pingcap/tidb/issues/29794) + TiCDC @@ -324,34 +330,34 @@ In v5.4, the key new features or improvements are as follows: + TiDB Data Migration (DM) - - Lower the usage rate of CPU when relay is enabled [#2214](https://github.com/pingcap/dm/issues/2214) + - Lower the usage rate of CPU when the relay is enabled [#2214](https://github.com/pingcap/dm/issues/2214) + TiDB Lightning - - Use optimistic transactions to write data to improve performance in TiDB-backend mode [#30953](https://github.com/pingcap/tidb/pull/30953) + - Use optimistic transactions by default to write data to improve performance in TiDB-backend mode [#30953](https://github.com/pingcap/tidb/pull/30953) + Dumpling - - Improve compatibility when Dumpling checks the data version [#29500](https://github.com/pingcap/tidb/pull/29500) - - Add a default collation when dumping `CREATE DATABASE` and `CREATE TABLE` [#3420](https://github.com/pingcap/tiflow/issues/3420) + - Improve compatibility when Dumpling checks the database version [#29500](https://github.com/pingcap/tidb/pull/29500) + - Add the default collation when dumping `CREATE DATABASE` and `CREATE TABLE` [#3420](https://github.com/pingcap/tiflow/issues/3420) ## Bug fixes + TiDB - Fix the issue of the `tidb_analyze_version` value change that occurs when upgrading the cluster from v4.x to v5.x [#25422](https://github.com/pingcap/tidb/issues/25422) - - Fix the issue of wrong result that occurs when using different collations in a subquery [#30748](https://github.com/pingcap/tidb/issues/30748) + - Fix the issue of the wrong result that occurs when using different collations in a subquery [#30748](https://github.com/pingcap/tidb/issues/30748) - Fix the issue that the result of `concat(ifnull(time(3))` in TiDB is different from that in MySQL [#29498](https://github.com/pingcap/tidb/issues/29498) - Fix the issue of potential data index inconsistency in optimistic transaction mode [#30410](https://github.com/pingcap/tidb/issues/30410) - Fix the issue that the query execution plan of IndexMerge is wrong when an expression cannot be pushed down to TiKV [#30200](https://github.com/pingcap/tidb/issues/30200) - Fix the issue that concurrent column type change causes inconsistency between the schema and the data [#31048](https://github.com/pingcap/tidb/issues/31048) - - Fix the issue that the IndexMerge query result is wrong that occurs when there is a subquery [#30913](https://github.com/pingcap/tidb/issues/30913) + - Fix the issue that the IndexMerge query result is wrong when there is a subquery [#30913](https://github.com/pingcap/tidb/issues/30913) - Fix the panic issue that occurs when the FetchSize is set too large in the client [#30896](https://github.com/pingcap/tidb/issues/30896) - Fix the issue that LEFT JOIN might be mistakenly converted to INNER JOIN [#20510](https://github.com/pingcap/tidb/issues/20510) - Fix the issue that panic might occur when the `CASE-WHEN` expression and collation are used together [#30245](https://github.com/pingcap/tidb/issues/30245) - Fix the issue of wrong query result that occurs when the `IN` value contains a binary constant [#31261](https://github.com/pingcap/tidb/issues/31261) - Fix the issue of wrong query result that occurs when CTE has a subquery [#31255](https://github.com/pingcap/tidb/issues/31255) - - Fix the issue that executing the `INSERT ... SELECT ... ON DUPLICATE KEY UPDATE` statement get panic [#28078](https://github.com/pingcap/tidb/issues/28078) + - Fix the issue that executing the `INSERT ... SELECT ... ON DUPLICATE KEY UPDATE` statement gets panic [#28078](https://github.com/pingcap/tidb/issues/28078) - Fix the issue that INDEX HASH JOIN returns the `send on closed channel` error [#31129](https://github.com/pingcap/tidb/issues/31129) + TiKV @@ -370,20 +376,14 @@ In v5.4, the key new features or improvements are as follows: + TiFlash - - Fix the issue that the `Tree struct based executor must have executor id` error is returned in the query result - - Fix the issue that the `Illegal column type UInt32 for the second argument of function tidbRoundWithFrac` error is returned in the query result - - Fix the issue that the `Unexpected type of column: Nullable(Nothing)` error is returned in the query result - - Fix the issue that the `source region at right may be considered at left when merging` error is returned in the query result - - Fix the issue that collation does not take effect on the `nullable` type - - Fix the issue that `coalesce` mistakenly deletes the `nullable` identifier from the result columns - - Fix the issue that TiFlash might panic when a MPP query is stopped - - Fix the issue that queries with the `where ` clause return wrong result + - Fix the issue that TiFlash might panic when an MPP query is stopped + - Fix the issue that queries with the `where ` clause return wrong results - Fix the potential issue of data inconsistency that might occur when setting the column type of an integer primary key to a larger range - Fix the issue that when an input time is earlier than 1970-01-01 00:00:01 UTC, the behavior of `unix_timestamp` is inconsistent with that of TiDB or MySQL - - Fix the issue that TiFlash might return the `EstablishMPPConnection` error after it is retarted - - Fix the issue that the `CastStringAsDecimal` bahavior is inconsitent in TiFlash and in TiDB/TiKV + - Fix the issue that TiFlash might return the `EstablishMPPConnection` error after it is restarted + - Fix the issue that the `CastStringAsDecimal` behavior is inconsistent in TiFlash and in TiDB/TiKV - Fix the issue that the `DB::Exception: Encode type of coprocessor response is not CHBlock` error is returned in the query result - - Fix the issue that the `castStringAsReal` bahavior is inconsitent in TiFlash and in TiDB/TiKV + - Fix the issue that the `castStringAsReal` behavior is inconsistent in TiFlash and in TiDB/TiKV - Fix the issue that the returned result of the `date_add_string_xxx` function in TiFlash is inconsistent with that in MySQL + Tools @@ -403,7 +403,7 @@ In v5.4, the key new features or improvements are as follows: - Fix the potential panic issue that occurs when a replication task is removed [#3128](https://github.com/pingcap/tiflow/issues/3128) - Fix the issue of panic and data inconsistency that occurs when outputting the default column value [#3929](https://github.com/pingcap/tiflow/issues/3929) - Fix the issue that default values cannot be replicated [#3793](https://github.com/pingcap/tiflow/issues/3793) - - Fix the potential issue that the deadlock causes a replication task to gets stuck [#4055](https://github.com/pingcap/tiflow/issues/4055) + - Fix the potential issue that the deadlock causes a replication task to get stuck [#4055](https://github.com/pingcap/tiflow/issues/4055) - Fix the issue that no log is output when the disk is fully written [#3362](https://github.com/pingcap/tiflow/issues/3362) - Fix the issue that special comments in DDL statements cause the replication task to stop [#3755](https://github.com/pingcap/tiflow/issues/3755) - Fix the issue that the service cannot be started because of a timezone issue in the RHEL release [#3584](https://github.com/pingcap/tiflow/issues/3584) @@ -416,12 +416,12 @@ In v5.4, the key new features or improvements are as follows: - Fix the issue that the `CREATE VIEW` statement interrupts data replication [#4173](https://github.com/pingcap/tiflow/issues/4173) - Fix the issue the schema needs to be reset after a DDL statement is skipped [#4177](https://github.com/pingcap/tiflow/issues/4177) - Fix the issue that the table checkpoint is not updated in time after a DDL statement is skipped [#4184](https://github.com/pingcap/tiflow/issues/4184) - - Fix a compatibility issue of the TiDB version with the parser version [#4298](https://github.com/pingcap/tiflow/issues/4298) + - Fix a compatibility issue of the TiDB version with the Parser version [#4298](https://github.com/pingcap/tiflow/issues/4298) - Fix the issue that syncer metrics are updated only when querying the status [#4281](https://github.com/pingcap/tiflow/issues/4281) + TiDB Lightning - - Fix the issue of wrong import result that occurs when TiDB Lightning does not have the priviledge to access the `mysql.tidb` table [#31088](https://github.com/pingcap/tidb/issues/31088) + - Fix the issue of wrong import result that occurs when TiDB Lightning does not have the privilege to access the `mysql.tidb` table [#31088](https://github.com/pingcap/tidb/issues/31088) - Fix the issue that some checks are skipped when TiDB Lightning is restarted [#30772](https://github.com/pingcap/tidb/issues/30772) - Fix the issue that TiDB Lighting fails to report the error when the S3 path does not exist [#30674](https://github.com/pingcap/tidb/pull/30674) From 01b8c9ba59a737ee32ace30703839a4ea249adea Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 21 Jan 2022 14:05:30 +0800 Subject: [PATCH 06/27] delete raft engine and add dm doc statement --- releases/release-5.4.0.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index 6df9bb8fee885..dc538287c5cdb 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -14,7 +14,6 @@ In v5.4, the key new features or improvements are as follows: - Support using Index Merge to access data, which merges the filtering results of indexes on multiple columns - Support reading stale data using a session variable - Support persisting the configuration for collecting statistics -- Support using Raft Engine as the log storage engine of TiKV (experimental) - Optimize the impact of backup on the cluster - Support using Azure Blob storage as the target storage for backup - Continuously improve the stability and performance of TiFlash and the MPP engine @@ -52,7 +51,6 @@ In v5.4, the key new features or improvements are as follows: | TiDB | [`stats-load-queue-size`](/tidb-configuration-file.md#stats-load-queue-size-new-in-v540) | Newly added | Controls the maximum number of column requests that the TiDB synchronously loading statistics feature can cache. The default value is`1000`. | | TiKV | [`snap-generator-pool-size`](/tidb-configuration-file.md#snap-generator-pool-size) | Newly added | The size of `snap-generator` thread pool. The default value is `2`. | | TiKV | `log.file.max-size`, `log.file.max-days`, `log.file.max-backups` | Newly added | For details, see [TiKV Configuration File - `log.file`](/tikv-configuration-file.md#logfile-new-in-v540). | -| TiKV | `raft-engine` | Newly added | Includes `enable`, `dir`, `batch-compression-threshold`, `bytes-per-sync`, `target-file-size`, `purge-threshold`, `recovery-mode`, `recovery-read-block-size`, `recovery-read-block-size`, and `recovery-threads`. For details, see [TiKV Configuration File - `raft-engine`](/tikv-configuration-file.md#raft-engine).| | TiKV | [`backup.enable-auto-tune`](/tikv-configuration-file.md#enable-auto-tune-new-in-v540) | Newly added | In v5.3.0, the default value is `false`. Since v5.4.0, the default value is changed to `true`. This parameter controls whether to limit the resources used by backup tasks to reduce the impact on the cluster when the cluster resource utilization is high. In the default configuration, the speed of backup tasks might slow down. | | TiKV | `log-level`, `log-format`, `log-file`, `log-rotation-size` | Modified | The names of TiKV log parameters are replaced with the names that are consistent with TiDB log parameters, which are `log.level`, `log.format`, `log.file.filename`, and `log.enable-timestamp`. If you only set the old parameters, and their values are set to non-default values, the old parameters remain compatible with the new parameters. If both old and new parameters are set, the new parameters take effect. For details, see [TiKV Configuration File - log](/tikv-configuration-file.md#log-new-in-v540). | | TiKV | `log-rotation-timespan` | Deleted | The timespan between log rotations. After this timespan passes, a log file is rotated, which means a timestamp is appended to the file name of the current log file, and a new log file is created. | @@ -81,6 +79,7 @@ In v5.4, the key new features or improvements are as follows: - An interface is added between TiDB and PD. When you use the `information_schema.TIDB_HOT_REGIONS_HISTORY` system table, TiDB needs to use PD in the corresponding version. - TiDB Server, PD Server, and TiKV Server start using a unified naming method for the log-related parameters to manage log names, output formats, and the rules for rotation and expiration. For details, see [TiKV configuration file - log](/tikv-configuration-file.md#log-new-in-v540). - Since v5.4.0, if you create a SQL binding for an execution plan that has been cached via Plan Cache, the binding invalidates the plan already cached for the corresponding query. The new binding does not affect execution plans cached before v5.4.0. +- In v5.3 and earlier versions, [TiDB Data Migration (DM)](https://docs.pingcap.com/tidb-data-migration/v5.3/) documentation is independent of TiDB documentation. Since v5.4, DM documentation is integrated into TiDB documentation with the same version. You can directly read [DM documentation](/dm/dm-overview.md) without accessing the DM documentation site. ## New features @@ -156,12 +155,6 @@ In v5.4, the key new features or improvements are as follows: [User document](/explain-index-merge.md) -- **Add Raft Engine (experimental)** - - Support using [Raft Engine](https://github.com/tikv/raft-engine) as the log storage engine in TiKV. Compared with RocksDB, Raft Engine can reduce TiKV I/O write traffic by up to 40% and CPU usage by 10%, while improving foreground throughput by about 5% and reducing tail latency by 20% under certain loads. In addition, Raft Engine improves the efficiency of log recycling and fixes the issue of log accumulation in extreme conditions. - - Raft Engine is still an experimental feature and is disabled by default. Note that the data format of Raft Engine in v5.4.0 is not compatible with previous versions. Before upgrading or downgrading the cluster, you need to make sure that Raft Engine on all TiKV nodes is disabled. It is recommended to use Raft Engine only in v5.4.0 or a later version. - - **Support collecting statistics on `PREDICATE COLUMNS` (experimental)** In most cases, when executing SQL statements, the optimizer only uses statistics of some columns (such as columns in the `WHERE`, `JOIN`, `ORDER BY`, and `GROUP BY` statements). These used columns are called `PREDICATE COLUMNS`. From 1c5f65afe6cc3b757b5f8752ef8b92ccb549429c Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 21 Jan 2022 14:35:24 +0800 Subject: [PATCH 07/27] Update experimental-features.md --- experimental-features.md | 1 - 1 file changed, 1 deletion(-) diff --git a/experimental-features.md b/experimental-features.md index 4234837051ad4..8931008b81dac 100644 --- a/experimental-features.md +++ b/experimental-features.md @@ -11,7 +11,6 @@ This document introduces the experimental features of TiDB in different versions + [Automatically scale TiFlash thread pool](/tiflash/tiflash-configuration.md). (Introduced in v5.4) + [Support synchronously loading statistics](/statistics.md#load-statistics). (Introduced in v5.4) -+ [Raft Engine](/tikv-configuration-file.md#raft-engine) (Introduced in v5.4) + [Support collecting statistics for `PREDICATE COLUMNS`](/statistics.md#collect-statistics-for-some-columns) (Introduced in v5.4) ## Stability From 9b0d72b0cde6f88332a6735309861bbac2176388 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 24 Jan 2022 09:54:40 +0800 Subject: [PATCH 08/27] Update releases/release-5.4.0.md Co-authored-by: sunzy --- releases/release-5.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index dc538287c5cdb..1f21455ad3b93 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -221,7 +221,7 @@ In v5.4, the key new features or improvements are as follows: - Recover the `enable-relay` switch in the `source` configuration. - - Support dynamically enabling or disabling relay log in the `start-relay` or `stop-relay` command. + - Support dynamically enabling and disabling relay log using the `start-relay` and `stop-relay` commands. - Bind the status of relay log to `source`. `source` keeps its original status of being enabled or disabled after it is migrated to any DM-worker. From 960c07ce86bee031674b6ea7e2797cbad116f8ec Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 24 Jan 2022 11:41:13 +0800 Subject: [PATCH 09/27] Apply suggestions from code review Co-authored-by: Ian --- releases/release-5.4.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index 1f21455ad3b93..69e56966cbd92 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -15,7 +15,7 @@ In v5.4, the key new features or improvements are as follows: - Support reading stale data using a session variable - Support persisting the configuration for collecting statistics - Optimize the impact of backup on the cluster -- Support using Azure Blob storage as the target storage for backup +- Support using Azure Blob storage as the backup storage - Continuously improve the stability and performance of TiFlash and the MPP engine - Add a switch in TiDB Lightning to determine whether to allow importing to an existing table with data - Optimize the Continuous Profiling feature (experimental) @@ -187,13 +187,13 @@ In v5.4, the key new features or improvements are as follows: - **Reduce the impact of backup tasks on the cluster** - Backup & Restore (BR) introduces the auto-tune feature (enabled by default). This feature can reduce the impact of backup tasks on the cluster by monitoring the cluster resource usage and adjusting the number of threads used by the backup tasks. In some cases, if you increase the cluster resource usage for backup and enable the auto-tune feature, it is possible to limit the impact of backup tasks on the cluster to 10% or less. + Backup & Restore (BR) introduces the auto-tune feature (enabled by default). This feature can reduce the impact of backup tasks on the cluster by monitoring the cluster resource usage and adjusting the number of threads used by the backup tasks. In some cases, if you increase the cluster hardware resource for backup and enable the auto-tune feature, it can limit the impact of backup tasks on the cluster to 10% or less. [User document](/br/br-auto-tune.md) - **Support Azure Blob Storage as a target storage for backup** - Backup & Restore (BR) supports Azure Blob Storage as a remote target storage for backup tasks. If you deploy TiDB in Azure Cloud, you can back up the cluster data to the Azure Blob Storage service. + Backup & Restore (BR) supports Azure Blob Storage as a remote backup storage. If you deploy TiDB in Azure Cloud, now you can back up the cluster data to the Azure Blob Storage service. [User document](/br/backup-and-restore-azblob.md) From 6f9d2c211e95c4d79b9a25aff9bd4c1e906034dd Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 24 Jan 2022 15:23:08 +0800 Subject: [PATCH 10/27] add back raft engine --- experimental-features.md | 3 ++- releases/release-5.4.0.md | 10 ++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/experimental-features.md b/experimental-features.md index 8931008b81dac..6b5e46ab7813c 100644 --- a/experimental-features.md +++ b/experimental-features.md @@ -10,8 +10,9 @@ This document introduces the experimental features of TiDB in different versions ## Performance + [Automatically scale TiFlash thread pool](/tiflash/tiflash-configuration.md). (Introduced in v5.4) -+ [Support synchronously loading statistics](/statistics.md#load-statistics). (Introduced in v5.4) ++ [Raft Engine](/tikv-configuration-file.md#raft-engine). (Introduced in v5.4) + [Support collecting statistics for `PREDICATE COLUMNS`](/statistics.md#collect-statistics-for-some-columns) (Introduced in v5.4) ++ [Support synchronously loading statistics](/statistics.md#load-statistics). (Introduced in v5.4) ## Stability diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index 69e56966cbd92..fbf306544ad73 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -14,6 +14,7 @@ In v5.4, the key new features or improvements are as follows: - Support using Index Merge to access data, which merges the filtering results of indexes on multiple columns - Support reading stale data using a session variable - Support persisting the configuration for collecting statistics +- Support using Raft Engine as the log storage engine of TiKV (experimental) - Optimize the impact of backup on the cluster - Support using Azure Blob storage as the backup storage - Continuously improve the stability and performance of TiFlash and the MPP engine @@ -51,6 +52,7 @@ In v5.4, the key new features or improvements are as follows: | TiDB | [`stats-load-queue-size`](/tidb-configuration-file.md#stats-load-queue-size-new-in-v540) | Newly added | Controls the maximum number of column requests that the TiDB synchronously loading statistics feature can cache. The default value is`1000`. | | TiKV | [`snap-generator-pool-size`](/tidb-configuration-file.md#snap-generator-pool-size) | Newly added | The size of `snap-generator` thread pool. The default value is `2`. | | TiKV | `log.file.max-size`, `log.file.max-days`, `log.file.max-backups` | Newly added | For details, see [TiKV Configuration File - `log.file`](/tikv-configuration-file.md#logfile-new-in-v540). | +| TiKV | `raft-engine` | Newly added | Includes `enable`, `dir`, `batch-compression-threshold`, `bytes-per-sync`, `target-file-size`, `purge-threshold`, `recovery-mode`, `recovery-read-block-size`, `recovery-read-block-size`, and `recovery-threads`. For details, see [TiKV Configuration File - `raft-engine`](/tikv-configuration-file.md#raft-engine).| | TiKV | [`backup.enable-auto-tune`](/tikv-configuration-file.md#enable-auto-tune-new-in-v540) | Newly added | In v5.3.0, the default value is `false`. Since v5.4.0, the default value is changed to `true`. This parameter controls whether to limit the resources used by backup tasks to reduce the impact on the cluster when the cluster resource utilization is high. In the default configuration, the speed of backup tasks might slow down. | | TiKV | `log-level`, `log-format`, `log-file`, `log-rotation-size` | Modified | The names of TiKV log parameters are replaced with the names that are consistent with TiDB log parameters, which are `log.level`, `log.format`, `log.file.filename`, and `log.enable-timestamp`. If you only set the old parameters, and their values are set to non-default values, the old parameters remain compatible with the new parameters. If both old and new parameters are set, the new parameters take effect. For details, see [TiKV Configuration File - log](/tikv-configuration-file.md#log-new-in-v540). | | TiKV | `log-rotation-timespan` | Deleted | The timespan between log rotations. After this timespan passes, a log file is rotated, which means a timestamp is appended to the file name of the current log file, and a new log file is created. | @@ -155,6 +157,14 @@ In v5.4, the key new features or improvements are as follows: [User document](/explain-index-merge.md) +- **Add Raft Engine (experimental)** + + Support using [Raft Engine](https://github.com/tikv/raft-engine) as the log storage engine in TiKV. Compared with RocksDB, Raft Engine can reduce TiKV I/O write traffic by up to 40% and CPU usage by 10%, while improving foreground throughput by about 5% and reducing tail latency by 20% under certain loads. In addition, Raft Engine improves the efficiency of log recycling and fixes the issue of log accumulation in extreme conditions. + + Raft Engine is still an experimental feature and is disabled by default. Note that the data format of Raft Engine in v5.4.0 is not compatible with previous versions. Before upgrading or downgrading the cluster, you need to make sure that Raft Engine on all TiKV nodes is disabled. It is recommended to use Raft Engine only in v5.4.0 or a later version. + + [User document](/tikv-configuration-file.md#raft-engine) + - **Support collecting statistics on `PREDICATE COLUMNS` (experimental)** In most cases, when executing SQL statements, the optimizer only uses statistics of some columns (such as columns in the `WHERE`, `JOIN`, `ORDER BY`, and `GROUP BY` statements). These used columns are called `PREDICATE COLUMNS`. From e7c3469afd0fc52c748878642787729838df4f93 Mon Sep 17 00:00:00 2001 From: qqqdan Date: Tue, 25 Jan 2022 15:28:14 +0800 Subject: [PATCH 11/27] Update releases/release-5.4.0.md Co-authored-by: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> --- releases/release-5.4.0.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index fbf306544ad73..7d0497e8bf513 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -105,6 +105,12 @@ In v5.4, the key new features or improvements are as follows: [User document](/tispark-overview.md#security) +- **TiUP support for generating an initial password for the root user** + + An `--init` parameter is introduced to the command for starting a cluster. With this parameter, in a TiDB cluster deployed using TiUP, an initial password is generated for the database root user. This avoids security risks in using a root user with an empty password and ensures the security of databases. + + [User document](/production-deployment-using-tiup.md##step-7-start-the-tidb-cluster) + ### Performance - **Continue improving the stability and performance of the columnar storage engine TiFlash and the computing engine MPP** From ce4efbbcc9fa75b121d816bd4820fff54cc88eec Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 26 Jan 2022 10:45:21 +0800 Subject: [PATCH 12/27] Apply suggestions from code review Co-authored-by: Grace Cai --- releases/release-5.4.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index 7d0497e8bf513..c9d24faf5247b 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -37,7 +37,7 @@ In v5.4, the key new features or improvements are as follows: | [`tidb_enable_top_sql`](/system-variables.md#tidb_enable_top_sql-new-in-v540) | Newly added | Controls whether to enable the Top SQL feature. The default value is `OFF`. | | [`tidb_persist_analyze_options`](/system-variables.md#tidb_persist_analyze_options-new-in-v540) | Newly added | Controls whether to enable the [ANALYZE configuration persistence](/statistics.md#persist-analyze-configurations) feature. The default value is `OFF`. | | [`tidb_read_staleness`](/system-variables.md#tidb_read_staleness-new-in-v540) | Newly added | Controls the range of historical data that can be read in the current session. The default value is `0`.| -| [`tidb_regard_null_as_point `](/system-variables.md#tidb_regard_null_as_point-new-in-v540) | Newly added | Controls whether the optimizer can treat `null` as a point value and uses it as a prefix condition to access the index. | +| [`tidb_regard_null_as_point `](/system-variables.md#tidb_regard_null_as_point-new-in-v540) | Newly added | Controls whether the optimizer can use a query condition including null equivalence as a prefix condition for index access. | | [`tidb_stats_load_sync_wait`](/system-variables.md#tidb_stats_load_sync_wait-new-in-v540) | Newly added | Controls whether to enable the synchronously loading statistics feature. The default value `0` means that the feature is disabled and that the statistics is asynchronously loaded. When the feature is enabled, this variable controls the maximum time that SQL optimization can wait for synchronously loading statistics before timeout. | | [`tidb_stats_load_pseudo_timeout`](/system-variables.md#tidb_stats_load_pseudo_timeout-new-in-v540) | Newly added | Controls when synchronously loading statistics reaches timeout, whether SQL fails (`OFF`) or falls back to using pseudo statistics (`ON`). The default value is `ON`. | | [`tidb_backoff_lock_fast`](/system-variables.md#tidb_backoff_lock_fast) | Modified | The default value is changed from `100` to `10`. | @@ -105,9 +105,9 @@ In v5.4, the key new features or improvements are as follows: [User document](/tispark-overview.md#security) -- **TiUP support for generating an initial password for the root user** +- **TiUP supports generating an initial password for the root user** - An `--init` parameter is introduced to the command for starting a cluster. With this parameter, in a TiDB cluster deployed using TiUP, an initial password is generated for the database root user. This avoids security risks in using a root user with an empty password and ensures the security of databases. + An `--init` parameter is introduced to the command for starting a cluster. With this parameter, in a TiDB cluster deployed using TiUP, TiUP generates an initial password for the database root user. This avoids security risks in using a root user with an empty password and ensures the security of databases. [User document](/production-deployment-using-tiup.md##step-7-start-the-tidb-cluster) From 827a9a93b5d224cd8ee8e171a71b604c0f4998f9 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 26 Jan 2022 10:49:28 +0800 Subject: [PATCH 13/27] fix link --- releases/release-5.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index c9d24faf5247b..c559bbb3c517c 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -231,7 +231,7 @@ In v5.4, the key new features or improvements are as follows: In Local-backend mode, TiDB Lightning outputs duplicated data before the data import is completed, and then removes that duplicated data from the database. You can resolve the duplicated data after the import is completed and select suitable data to insert according to application rules. It is recommended to clean upstream data sources based on duplicated data to avoid data inconsistency caused by duplicated data encountered in the subsequent incremental data migration phase. - [User document](tidb-lightning/tidb-lightning-error-resolution.md) + [User document](/tidb-lightning/tidb-lightning-error-resolution.md) - **Optimize the usage of relay log in TiDB Data Migration (DM)** From 5d90ae78c07033838dd5960c65b07f3dcb2d71de Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 26 Jan 2022 10:51:00 +0800 Subject: [PATCH 14/27] fix jenkins --- releases/release-5.4.0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index c559bbb3c517c..f08f26e68abba 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -37,7 +37,7 @@ In v5.4, the key new features or improvements are as follows: | [`tidb_enable_top_sql`](/system-variables.md#tidb_enable_top_sql-new-in-v540) | Newly added | Controls whether to enable the Top SQL feature. The default value is `OFF`. | | [`tidb_persist_analyze_options`](/system-variables.md#tidb_persist_analyze_options-new-in-v540) | Newly added | Controls whether to enable the [ANALYZE configuration persistence](/statistics.md#persist-analyze-configurations) feature. The default value is `OFF`. | | [`tidb_read_staleness`](/system-variables.md#tidb_read_staleness-new-in-v540) | Newly added | Controls the range of historical data that can be read in the current session. The default value is `0`.| -| [`tidb_regard_null_as_point `](/system-variables.md#tidb_regard_null_as_point-new-in-v540) | Newly added | Controls whether the optimizer can use a query condition including null equivalence as a prefix condition for index access. | +| [`tidb_regard_null_as_point`](/system-variables.md#tidb_regard_null_as_point-new-in-v540) | Newly added | Controls whether the optimizer can use a query condition including null equivalence as a prefix condition for index access. | | [`tidb_stats_load_sync_wait`](/system-variables.md#tidb_stats_load_sync_wait-new-in-v540) | Newly added | Controls whether to enable the synchronously loading statistics feature. The default value `0` means that the feature is disabled and that the statistics is asynchronously loaded. When the feature is enabled, this variable controls the maximum time that SQL optimization can wait for synchronously loading statistics before timeout. | | [`tidb_stats_load_pseudo_timeout`](/system-variables.md#tidb_stats_load_pseudo_timeout-new-in-v540) | Newly added | Controls when synchronously loading statistics reaches timeout, whether SQL fails (`OFF`) or falls back to using pseudo statistics (`ON`). The default value is `ON`. | | [`tidb_backoff_lock_fast`](/system-variables.md#tidb_backoff_lock_fast) | Modified | The default value is changed from `100` to `10`. | @@ -329,7 +329,7 @@ In v5.4, the key new features or improvements are as follows: + Backup & Restore (BR) - - Add a validity check for the key when BR performs encrypted backup [#29794](https://github.com/pingcap/tidb/issues/29794) + - Add a validity check for the key when BR performs encrypted backup [#29794](https://github.com/pingcap/tidb/issues/29794) + TiCDC From 2208ecbcbfda5a9391a7b00430126890b7404e1d Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 26 Jan 2022 11:08:26 +0800 Subject: [PATCH 15/27] fix --- releases/release-5.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index f08f26e68abba..ebf11b2001d5d 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -68,7 +68,7 @@ In v5.4, the key new features or improvements are as follows: | TiDB Data Migration (DM) | [`collation_compatible`](/dm/task-configuration-file-full.md#task-configuration-file-template-advanced) | Newly added | The mode to sync the default collation in `CREATE` SQL statements. The value options are "loose" (by default) and "strict". | | TiCDC | `max-message-bytes` | Modified| Change the default value of `max-message-bytes` in Kafka sink to `104857601` (10MB) | | TiCDC | `partition-num` | Modified | Change the default value of `partition-num` in Kafka Sink from `4` to `3`. It makes TiCDC send messages to Kafaka partitions more evenly. | -| TiDB Lightning | `meta-schema-name` | Modified | Specifies the schema name for the metadata in the target TiDB. From v5.4.0, this schema is created only if you have enabled [parallel import](/tidb-lightning/tidb-lightning-distributed-import) (the corresponding parameter is `tikv-importer.incremental-import = true`). | +| TiDB Lightning | `meta-schema-name` | Modified | Specifies the schema name for the metadata in the target TiDB. From v5.4.0, this schema is created only if you have enabled [parallel import](/tidb-lightning/tidb-lightning-distributed-import.md) (the corresponding parameter is `tikv-importer.incremental-import = true`). | | TiDB Lightning | `task-info-schema-name` | Newly added | Specifies the name of the database where duplicated data is stored when TiDB Lightning detects conflicts. By default, the value is "lightning_task_info". Specify this parameter only if you have enabled the "duplicate-resolution" feature. | | TiDB Lightning | `incremental-import` | Newly added | Determines whether to allow importing data to tables where data already exists. The default value is `false`. | From f2679c0e90614edadd7a683bce3adf55abaee0b5 Mon Sep 17 00:00:00 2001 From: shichun-0415 <89768198+shichun-0415@users.noreply.github.com> Date: Wed, 26 Jan 2022 13:53:13 +0800 Subject: [PATCH 16/27] Apply suggestions from code review --- releases/release-5.4.0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index ebf11b2001d5d..546b8e495a91d 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -107,9 +107,9 @@ In v5.4, the key new features or improvements are as follows: - **TiUP supports generating an initial password for the root user** - An `--init` parameter is introduced to the command for starting a cluster. With this parameter, in a TiDB cluster deployed using TiUP, TiUP generates an initial password for the database root user. This avoids security risks in using a root user with an empty password and ensures the security of databases. + An `--init` parameter is introduced to the command for starting a cluster. With this parameter, in a TiDB cluster deployed using TiUP, TiUP generates an initial strong password for the database root user. This avoids security risks in using a root user with an empty password and ensures the security of databases. - [User document](/production-deployment-using-tiup.md##step-7-start-the-tidb-cluster) + [User document](/production-deployment-using-tiup.md#step-7-start-the-tidb-cluster) ### Performance From 8414814b41a334cc58ebd0e2742891ad601ce62c Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 26 Jan 2022 14:22:25 +0800 Subject: [PATCH 17/27] fix broken anchors --- releases/release-5.4.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index 546b8e495a91d..5cd514d7f7762 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -42,7 +42,7 @@ In v5.4, the key new features or improvements are as follows: | [`tidb_stats_load_pseudo_timeout`](/system-variables.md#tidb_stats_load_pseudo_timeout-new-in-v540) | Newly added | Controls when synchronously loading statistics reaches timeout, whether SQL fails (`OFF`) or falls back to using pseudo statistics (`ON`). The default value is `ON`. | | [`tidb_backoff_lock_fast`](/system-variables.md#tidb_backoff_lock_fast) | Modified | The default value is changed from `100` to `10`. | | [`tidb_enable_index_merge`](/system-variables.md#tidb_enable_index_merge-new-in-v40) | Modified | The default value is changed from `OFF` to `ON`.
  • If you upgrade a TiDB cluster from versions earlier than v4.0.0 to v5.4.0 or later, this variable is `OFF` by default.
  • If you upgrade a TiDB cluster from v4.0.0 or later to v5.4.0 or later, this variable remains the same as before the upgrade.
  • For the newly created TiDB clusters of v5.4.0 and later, this variable is `ON` by default.
| -| [`tidb_store_limit`](/system-variables.md#tidb_store_limit-new-i-v304-and-v40) | Modified | Before v5.4.0, this variable can be configured at instance level and globally. Starting from v5.4.0, this variable only supports global configuration. | +| [`tidb_store_limit`](/system-variables.md#tidb_store_limit-new-in-v304-and-v40) | Modified | Before v5.4.0, this variable can be configured at instance level and globally. Starting from v5.4.0, this variable only supports global configuration. | ### Configuration file parameters @@ -175,11 +175,11 @@ In v5.4, the key new features or improvements are as follows: In most cases, when executing SQL statements, the optimizer only uses statistics of some columns (such as columns in the `WHERE`, `JOIN`, `ORDER BY`, and `GROUP BY` statements). These used columns are called `PREDICATE COLUMNS`. - Since v5.4.0, you can set the value of the [`tidb_enable_column_tracking`](/system-variables.md#tidb_enable_column_tracking-introduced-from-v540-version) system variable to `ON` to enable TiDB to collect `PREDICATE COLUMNS`. + Since v5.4.0, you can set the value of the [`tidb_enable_column_tracking`](/system-variables.md#tidb_enable_column_tracking-new-in-v540) system variable to `ON` to enable TiDB to collect `PREDICATE COLUMNS`. After the setting, TiDB writes the `PREDICATE COLUMNS` information to the `mysql.column_stats_usage` system table every 100 * [`stats-lease`](/tidb-configuration-file.md#stats-lease). When the query pattern of your business is stable, you can use the `ANALYZE TABLE TableName PREDICATE COLUMNS` syntax to collect statistics on the `PREDICATE COLUMNS` columns only, which can greatly reduce the overhead of collecting statistics. - [User document](/statistics.md#collect-statistics-for-some-columns) + [User document](/statistics.md#collect-statistics-on-some-columns) - **Support synchronously loading statistics (experimental)** From 66a155fbf71fd91788ee3ea01c2ce05d1d3c694e Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 26 Jan 2022 14:29:39 +0800 Subject: [PATCH 18/27] add snap-generator-pool-size --- releases/release-5.4.0.md | 2 +- tikv-configuration-file.md | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index 5cd514d7f7762..d745bece391c0 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -50,7 +50,7 @@ In v5.4, the key new features or improvements are as follows: | :---------- | :----------- | :----------- | :----------- | | TiDB | [`stats-load-concurrency`](/tidb-configuration-file.md#stats-load-concurrency-new-in-v540) | Newly added | Controls the maximum number of columns that the TiDB synchronously loading statistics feature can process concurrently. The default value is `5`. | | TiDB | [`stats-load-queue-size`](/tidb-configuration-file.md#stats-load-queue-size-new-in-v540) | Newly added | Controls the maximum number of column requests that the TiDB synchronously loading statistics feature can cache. The default value is`1000`. | -| TiKV | [`snap-generator-pool-size`](/tidb-configuration-file.md#snap-generator-pool-size) | Newly added | The size of `snap-generator` thread pool. The default value is `2`. | +| TiKV | [`snap-generator-pool-size`](/tikv-configuration-file.md#snap-generator-pool-size-new-in-v540) | Newly added | The size of `snap-generator` thread pool. The default value is `2`. | | TiKV | `log.file.max-size`, `log.file.max-days`, `log.file.max-backups` | Newly added | For details, see [TiKV Configuration File - `log.file`](/tikv-configuration-file.md#logfile-new-in-v540). | | TiKV | `raft-engine` | Newly added | Includes `enable`, `dir`, `batch-compression-threshold`, `bytes-per-sync`, `target-file-size`, `purge-threshold`, `recovery-mode`, `recovery-read-block-size`, `recovery-read-block-size`, and `recovery-threads`. For details, see [TiKV Configuration File - `raft-engine`](/tikv-configuration-file.md#raft-engine).| | TiKV | [`backup.enable-auto-tune`](/tikv-configuration-file.md#enable-auto-tune-new-in-v540) | Newly added | In v5.3.0, the default value is `false`. Since v5.4.0, the default value is changed to `true`. This parameter controls whether to limit the resources used by backup tasks to reduce the impact on the cluster when the cluster resource utilization is high. In the default configuration, the speed of backup tasks might slow down. | diff --git a/tikv-configuration-file.md b/tikv-configuration-file.md index 3f7f09841c6cb..69e393658d051 100644 --- a/tikv-configuration-file.md +++ b/tikv-configuration-file.md @@ -608,6 +608,13 @@ Configuration items related to Raftstore. + Default value: `"4h"` + Minimum value: `0` +## `snap-generator-pool-size` New in v5.4.0 + ++ Configures the size of the `snap-generator` thread pool. ++ To make Regions generate snapshot faster in TiKV in recovery scenarios, you need to increase the count of the `snap-generator` threads of the corresponding worker. You can use this configuration item to increase the size of the `snap-generator` thread pool. ++ Default value: `2` ++ Minimum value: `0` + ### `lock-cf-compact-interval` + The time interval at which TiKV triggers a manual compaction for the Lock Column Family From 20d8c5f86999e99c8abd499802e165dbddacb333 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 26 Jan 2022 14:31:21 +0800 Subject: [PATCH 19/27] fix anchors --- experimental-features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/experimental-features.md b/experimental-features.md index 6b5e46ab7813c..87088887807a2 100644 --- a/experimental-features.md +++ b/experimental-features.md @@ -11,7 +11,7 @@ This document introduces the experimental features of TiDB in different versions + [Automatically scale TiFlash thread pool](/tiflash/tiflash-configuration.md). (Introduced in v5.4) + [Raft Engine](/tikv-configuration-file.md#raft-engine). (Introduced in v5.4) -+ [Support collecting statistics for `PREDICATE COLUMNS`](/statistics.md#collect-statistics-for-some-columns) (Introduced in v5.4) ++ [Support collecting statistics for `PREDICATE COLUMNS`](/statistics.md#collect-statistics-on-some-columns) (Introduced in v5.4) + [Support synchronously loading statistics](/statistics.md#load-statistics). (Introduced in v5.4) ## Stability From 38f7395796faf7f47611f83b0765017bff8308be Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Wed, 26 Jan 2022 14:32:28 +0800 Subject: [PATCH 20/27] bump version for pd --- pd-control.md | 2 +- pd-recover.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pd-control.md b/pd-control.md index 909d3e8e3497e..d9c909ce465c8 100644 --- a/pd-control.md +++ b/pd-control.md @@ -27,7 +27,7 @@ If you want to download the latest version of `pd-ctl`, directly download the Ti > **Note:** > -> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.3.0`, the package download link is `https://download.pingcap.org/tidb-v5.3.0-linux-amd64.tar.gz`. +> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.4.0`, the package download link is `https://download.pingcap.org/tidb-v5.4.0-linux-amd64.tar.gz`. ### Compile from source code diff --git a/pd-recover.md b/pd-recover.md index 920778bbb0173..355430ac733d1 100644 --- a/pd-recover.md +++ b/pd-recover.md @@ -26,7 +26,7 @@ To download the latest version of PD Recover, directly download the TiDB package > **Note:** > -> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.3.0`, the package download link is `https://download.pingcap.org/tidb-v5.3.0-linux-amd64.tar.gz`. +> `{version}` indicates the version number of TiDB. For example, if `{version}` is `v5.4.0`, the package download link is `https://download.pingcap.org/tidb-v5.4.0-linux-amd64.tar.gz`. ## Quick Start From a4c2de7db1b3f6ba078cd7a260df254fdba44ef9 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 28 Jan 2022 15:49:38 +0800 Subject: [PATCH 21/27] remove tidb dashboard root --- releases/release-5.4.0.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index d745bece391c0..62f46ef0dd23c 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -74,10 +74,6 @@ In v5.4, the key new features or improvements are as follows: ### Others -- TiDB Dashboard, by default, does not permit using `root` and empty password for login. - - Since v5.4.0, it is recommended to use `start --initial` when you use TiUP to start a TiDB cluster. Then, a random password is generated for the `root` account to log into TiDB Dashboard. - - An interface is added between TiDB and PD. When you use the `information_schema.TIDB_HOT_REGIONS_HISTORY` system table, TiDB needs to use PD in the corresponding version. - TiDB Server, PD Server, and TiKV Server start using a unified naming method for the log-related parameters to manage log names, output formats, and the rules for rotation and expiration. For details, see [TiKV configuration file - log](/tikv-configuration-file.md#log-new-in-v540). - Since v5.4.0, if you create a SQL binding for an execution plan that has been cached via Plan Cache, the binding invalidates the plan already cached for the corresponding query. The new binding does not affect execution plans cached before v5.4.0. From 2882b241a5e152098cb8b063b43f168eab225d29 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 28 Jan 2022 15:58:11 +0800 Subject: [PATCH 22/27] fix anchor --- releases/release-5.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index 62f46ef0dd23c..03e7b36b3182a 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -105,7 +105,7 @@ In v5.4, the key new features or improvements are as follows: An `--init` parameter is introduced to the command for starting a cluster. With this parameter, in a TiDB cluster deployed using TiUP, TiUP generates an initial strong password for the database root user. This avoids security risks in using a root user with an empty password and ensures the security of databases. - [User document](/production-deployment-using-tiup.md#step-7-start-the-tidb-cluster) + [User document](/production-deployment-using-tiup.md#step-7-start-a-tidb-cluster) ### Performance From a41c7d61e48b3b964aff1de90a0d4269fd660c09 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 7 Feb 2022 13:56:35 +0800 Subject: [PATCH 23/27] mention the removal of pitr and cdclog --- releases/release-5.4.0.md | 1 + 1 file changed, 1 insertion(+) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index 03e7b36b3182a..cff5bcd36f4ee 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -78,6 +78,7 @@ In v5.4, the key new features or improvements are as follows: - TiDB Server, PD Server, and TiKV Server start using a unified naming method for the log-related parameters to manage log names, output formats, and the rules for rotation and expiration. For details, see [TiKV configuration file - log](/tikv-configuration-file.md#log-new-in-v540). - Since v5.4.0, if you create a SQL binding for an execution plan that has been cached via Plan Cache, the binding invalidates the plan already cached for the corresponding query. The new binding does not affect execution plans cached before v5.4.0. - In v5.3 and earlier versions, [TiDB Data Migration (DM)](https://docs.pingcap.com/tidb-data-migration/v5.3/) documentation is independent of TiDB documentation. Since v5.4, DM documentation is integrated into TiDB documentation with the same version. You can directly read [DM documentation](/dm/dm-overview.md) without accessing the DM documentation site. +- Remove the experimental feature of Point-in-time recovery (PITR) along with cdclog. Since v5.4.0, PITR and cdclog are no longer supported. ## New features From 3d12abe0a1544c6340e591de285b3a48da02a74b Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 7 Feb 2022 16:06:13 +0800 Subject: [PATCH 24/27] address comment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Daniël van Eeden --- releases/release-5.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index cff5bcd36f4ee..5614c10cbe084 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -78,7 +78,7 @@ In v5.4, the key new features or improvements are as follows: - TiDB Server, PD Server, and TiKV Server start using a unified naming method for the log-related parameters to manage log names, output formats, and the rules for rotation and expiration. For details, see [TiKV configuration file - log](/tikv-configuration-file.md#log-new-in-v540). - Since v5.4.0, if you create a SQL binding for an execution plan that has been cached via Plan Cache, the binding invalidates the plan already cached for the corresponding query. The new binding does not affect execution plans cached before v5.4.0. - In v5.3 and earlier versions, [TiDB Data Migration (DM)](https://docs.pingcap.com/tidb-data-migration/v5.3/) documentation is independent of TiDB documentation. Since v5.4, DM documentation is integrated into TiDB documentation with the same version. You can directly read [DM documentation](/dm/dm-overview.md) without accessing the DM documentation site. -- Remove the experimental feature of Point-in-time recovery (PITR) along with cdclog. Since v5.4.0, PITR and cdclog are no longer supported. +- Remove the experimental feature of Point-in-time recovery (PITR) along with cdclog. Since v5.4.0, cdclog-based PITR and cdclog are no longer supported. ## New features From 1ab2146f389945fcd578319e62ce5d5a6ffda4f1 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Mon, 14 Feb 2022 11:59:52 +0800 Subject: [PATCH 25/27] minor wording change --- releases/release-5.4.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index 5614c10cbe084..b743381bead19 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -26,7 +26,7 @@ In v5.4, the key new features or improvements are as follows: > **Note:** > -> When upgrading from an earlier TiDB version to v5.3.0, if you want to know the compatibility change notes of all intermediate versions, you can check the [Release Notes](/releases/release-notes.md) of the corresponding version. +> When upgrading from an earlier TiDB version to v5.4.0, if you want to know the compatibility change notes of all intermediate versions, you can check the [Release Notes](/releases/release-notes.md) of the corresponding version. ### System variables @@ -50,7 +50,7 @@ In v5.4, the key new features or improvements are as follows: | :---------- | :----------- | :----------- | :----------- | | TiDB | [`stats-load-concurrency`](/tidb-configuration-file.md#stats-load-concurrency-new-in-v540) | Newly added | Controls the maximum number of columns that the TiDB synchronously loading statistics feature can process concurrently. The default value is `5`. | | TiDB | [`stats-load-queue-size`](/tidb-configuration-file.md#stats-load-queue-size-new-in-v540) | Newly added | Controls the maximum number of column requests that the TiDB synchronously loading statistics feature can cache. The default value is`1000`. | -| TiKV | [`snap-generator-pool-size`](/tikv-configuration-file.md#snap-generator-pool-size-new-in-v540) | Newly added | The size of `snap-generator` thread pool. The default value is `2`. | +| TiKV | [`snap-generator-pool-size`](/tikv-configuration-file.md#snap-generator-pool-size-new-in-v540) | Newly added | The size of the `snap-generator` thread pool. The default value is `2`. | | TiKV | `log.file.max-size`, `log.file.max-days`, `log.file.max-backups` | Newly added | For details, see [TiKV Configuration File - `log.file`](/tikv-configuration-file.md#logfile-new-in-v540). | | TiKV | `raft-engine` | Newly added | Includes `enable`, `dir`, `batch-compression-threshold`, `bytes-per-sync`, `target-file-size`, `purge-threshold`, `recovery-mode`, `recovery-read-block-size`, `recovery-read-block-size`, and `recovery-threads`. For details, see [TiKV Configuration File - `raft-engine`](/tikv-configuration-file.md#raft-engine).| | TiKV | [`backup.enable-auto-tune`](/tikv-configuration-file.md#enable-auto-tune-new-in-v540) | Newly added | In v5.3.0, the default value is `false`. Since v5.4.0, the default value is changed to `true`. This parameter controls whether to limit the resources used by backup tasks to reduce the impact on the cluster when the cluster resource utilization is high. In the default configuration, the speed of backup tasks might slow down. | @@ -160,7 +160,7 @@ In v5.4, the key new features or improvements are as follows: [User document](/explain-index-merge.md) -- **Add Raft Engine (experimental)** +- **Support Raft Engine (experimental)** Support using [Raft Engine](https://github.com/tikv/raft-engine) as the log storage engine in TiKV. Compared with RocksDB, Raft Engine can reduce TiKV I/O write traffic by up to 40% and CPU usage by 10%, while improving foreground throughput by about 5% and reducing tail latency by 20% under certain loads. In addition, Raft Engine improves the efficiency of log recycling and fixes the issue of log accumulation in extreme conditions. From 417671589bab59612fdd954f036d522024ee877c Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Tue, 15 Feb 2022 09:53:41 +0800 Subject: [PATCH 26/27] change date --- releases/release-5.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-5.4.0.md b/releases/release-5.4.0.md index b743381bead19..0f61c31655175 100644 --- a/releases/release-5.4.0.md +++ b/releases/release-5.4.0.md @@ -4,7 +4,7 @@ title: TiDB 5.4 Release Notes # TiDB 5.4 Release Notes -Release date：February xx, 2022 +Release date：February 15, 2022 TiDB version: 5.4.0 From 97d5e105ac5a2e78b6ee8e69c89eeb42701ac297 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Tue, 15 Feb 2022 10:24:23 +0800 Subject: [PATCH 27/27] Update releases/release-timeline.md --- releases/release-timeline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-timeline.md b/releases/release-timeline.md index 4d10acbad4b56..6bb121fdb8d4e 100644 --- a/releases/release-timeline.md +++ b/releases/release-timeline.md @@ -9,7 +9,7 @@ This document shows all the released TiDB versions in reverse chronological orde | Version | Release Date | | :--- | :--- | -| [5.4.0](/releases/release-5.4.0.md) | 2022-x-x | +| [5.4.0](/releases/release-5.4.0.md) | 2022-02-15 | | [5.0.6](/releases/release-5.0.6.md) | 2021-12-31 | | [4.0.16](/releases/release-4.0.16.md) | 2021-12-17 | | [5.1.3](/releases/release-5.1.3.md) | 2021-12-03 |