From 46e23047dce3633ca026a452dd4fb62edfc6016c Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Wed, 4 Sep 2024 17:35:43 +0800 Subject: [PATCH 01/11] Add temp.md --- temp.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 temp.md diff --git a/temp.md b/temp.md new file mode 100644 index 0000000000000..af27ff4986a7b --- /dev/null +++ b/temp.md @@ -0,0 +1 @@ +This is a test file. \ No newline at end of file From 41a0e741e2eaae0e54d5f31e076cb85867495d2f Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Wed, 4 Sep 2024 17:35:48 +0800 Subject: [PATCH 02/11] Delete temp.md --- temp.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 temp.md diff --git a/temp.md b/temp.md deleted file mode 100644 index af27ff4986a7b..0000000000000 --- a/temp.md +++ /dev/null @@ -1 +0,0 @@ -This is a test file. \ No newline at end of file From 64ab15ef0abcd496ccb97e6c9802f074b57cfc06 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Thu, 14 Nov 2024 18:17:25 +0800 Subject: [PATCH 03/11] Make wording consistent --- partitioned-table.md | 6 +++--- releases/release-6.3.0.md | 4 ++-- releases/release-6.5.10.md | 2 +- releases/release-7.3.0.md | 2 +- releases/release-7.5.0.md | 2 +- releases/release-8.1.1.md | 2 +- releases/release-8.2.0.md | 2 +- sql-statements/sql-statement-drop-stats.md | 4 ++-- statistics.md | 14 +++++++------- system-variables.md | 8 ++++---- tiflash-upgrade-guide.md | 2 +- 11 files changed, 24 insertions(+), 24 deletions(-) diff --git a/partitioned-table.md b/partitioned-table.md index e3140b633ba3a..42da76865b252 100644 --- a/partitioned-table.md +++ b/partitioned-table.md @@ -1926,7 +1926,7 @@ This variable is only used in table creation. After the table is created, modify ### Dynamic pruning mode -TiDB accesses partitioned tables in either `dynamic` or `static` mode. `dynamic` mode is used by default since v6.3.0. However, dynamic partitioning is effective only after the full table-level statistics, or GlobalStats, are collected. Before GlobalStats are collected, TiDB will use the `static` mode instead. For detailed information about GlobalStats, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). +TiDB accesses partitioned tables in either `dynamic` or `static` mode. `dynamic` mode is used by default since v6.3.0. However, dynamic partitioning is effective only after the full table-level statistics, or global statistics, are collected. If the `dynamic` mode is selected but the collection of global statistics is not completed, TiDB will use the `static` mode instead. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). {{< copyable "sql" >}} @@ -1936,9 +1936,9 @@ set @@session.tidb_partition_prune_mode = 'dynamic' Manual ANALYZE and normal queries use the session-level `tidb_partition_prune_mode` setting. The `auto-analyze` operation in the background uses the global `tidb_partition_prune_mode` setting. -In `static` mode, partitioned tables use partition-level statistics. In `dynamic` mode, partitioned tables use table-level GlobalStats. +In `static` mode, partitioned tables use partition-level statistics. In `dynamic` mode, partitioned tables use table-level global statistics. -When switching from `static` mode to `dynamic` mode, you need to check and collect statistics manually. This is because after the switch to `dynamic` mode, partitioned tables have only partition-level statistics but no table-level statistics. GlobalStats are collected only upon the next `auto-analyze` operation. +When switching from `static` mode to `dynamic` mode, you need to check and collect statistics manually. This is because after the switch to `dynamic` mode, partitioned tables have only partition-level statistics but no table-level statistics. Global statistics are collected only upon the next `auto-analyze` operation. {{< copyable "sql" >}} diff --git a/releases/release-6.3.0.md b/releases/release-6.3.0.md index f036afaa18a55..85b15fe0c4790 100644 --- a/releases/release-6.3.0.md +++ b/releases/release-6.3.0.md @@ -145,9 +145,9 @@ In v6.3.0-DMR, the key new features and improvements are as follows: You can [disable Titan](/storage-engine/titan-configuration.md#disable-titan) for online TiKV nodes. -* Use `static` partition pruning when GlobalStats are not ready [#37535](https://github.com/pingcap/tidb/issues/37535) @[Yisaer](https://github.com/Yisaer) +* Use `static` partition pruning when global statistics are not ready [#37535](https://github.com/pingcap/tidb/issues/37535) @[Yisaer](https://github.com/Yisaer) - When [`dynamic pruning`](/partitioned-table.md#dynamic-pruning-mode) is enabled, the optimizer selects execution plans based on [GlobalStats](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). Before GlobalStats are fully collected, using pseudo statistics might cause performance regression. In v6.3.0, this issue is addressed by maintaining the `static` mode if you enable dynamic pruning before GlobalStats are collected. TiDB remains in the `static` mode until GlobalStats are collected. This ensures performance stability when you change the partition pruning settings. + When [`dynamic pruning`](/partitioned-table.md#dynamic-pruning-mode) is enabled, the optimizer selects execution plans based on [GlobalStats](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). Before global statistics are fully collected, using pseudo statistics might cause performance regression. In v6.3.0, this issue is addressed by maintaining the `static` mode if you enable dynamic pruning before global statistics are collected. TiDB remains in the `static` mode until global statistics are collected. This ensures performance stability when you change the partition pruning settings. ### Ease of use diff --git a/releases/release-6.5.10.md b/releases/release-6.5.10.md index 9971b6c14ab4d..c5587abbdbc55 100644 --- a/releases/release-6.5.10.md +++ b/releases/release-6.5.10.md @@ -52,7 +52,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v6.5/quick-start-with- - Fix the issue that the `STATE` field in the `INFORMATION_SCHEMA.TIDB_TRX` table is empty due to the `size` of the `STATE` field not being defined [#53026](https://github.com/pingcap/tidb/issues/53026) @[cfzjywxk](https://github.com/cfzjywxk) - Fix the issue that TiDB does not create corresponding statistics metadata (`stats_meta`) when creating a table with foreign keys [#53652](https://github.com/pingcap/tidb/issues/53652) @[hawkingrei](https://github.com/hawkingrei) - Fix the issue that the statistics synchronous loading mechanism might fail unexpectedly under high query concurrency [#52294](https://github.com/pingcap/tidb/issues/52294) @[hawkingrei](https://github.com/hawkingrei) - - Fix the issue that the `Distinct_count` information in GlobalStats might be incorrect [#53752](https://github.com/pingcap/tidb/issues/53752) @[hawkingrei](https://github.com/hawkingrei) + - Fix the issue that the `Distinct_count` information in global statistics might be incorrect [#53752](https://github.com/pingcap/tidb/issues/53752) @[hawkingrei](https://github.com/hawkingrei) - Fix the issue that the histogram and TopN in the primary key column statistics are not loaded after restarting TiDB [#37548](https://github.com/pingcap/tidb/issues/37548) @[hawkingrei](https://github.com/hawkingrei) - Fix the issue that certain filter conditions in queries might cause the planner module to report an `invalid memory address or nil pointer dereference` error [#53582](https://github.com/pingcap/tidb/issues/53582) [#53580](https://github.com/pingcap/tidb/issues/53580) [#53594](https://github.com/pingcap/tidb/issues/53594) [#53603](https://github.com/pingcap/tidb/issues/53603) @[YangKeao](https://github.com/YangKeao) - Fix the issue that `PREPARE`/`EXECUTE` statements with the `CONV` expression containing a `?` argument might result in incorrect query results when executed multiple times [#53505](https://github.com/pingcap/tidb/issues/53505) @[qw4990](https://github.com/qw4990) diff --git a/releases/release-7.3.0.md b/releases/release-7.3.0.md index fa8b53a66cfb9..258e7078012b8 100644 --- a/releases/release-7.3.0.md +++ b/releases/release-7.3.0.md @@ -182,7 +182,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v7.3/quick-start-with- | [`tidb_allow_tiflash_cop`](/system-variables.md#tidb_allow_tiflash_cop-new-in-v730) | Newly added | This system variable is used to select the protocol for generating execution plans when TiDB pushes computation tasks down to TiFlash. | | [`tidb_lock_unchanged_keys`](/system-variables.md#tidb_lock_unchanged_keys-new-in-v711-and-v730) | Newly added | This variable is used to control in certain scenarios whether to lock the keys that are involved but not modified in a transaction. | | [`tidb_opt_enable_non_eval_scalar_subquery`](/system-variables.md#tidb_opt_enable_non_eval_scalar_subquery-new-in-v730) | Newly added | Controls whether the `EXPLAIN` statement disables the execution of constant subqueries that can be expanded at the optimization stage. | -| [`tidb_skip_missing_partition_stats`](/system-variables.md#tidb_skip_missing_partition_stats-new-in-v730) | Newly added | This variable controls the generation of GlobalStats when partition statistics are missing. | +| [`tidb_skip_missing_partition_stats`](/system-variables.md#tidb_skip_missing_partition_stats-new-in-v730) | Newly added | This variable controls the generation of global statistics when partition statistics are missing. | | [`tiflash_replica_read`](/system-variables.md#tiflash_replica_read-new-in-v730) | Newly added | Controls the strategy for selecting TiFlash replicas when a query requires the TiFlash engine. | ### Configuration file parameters diff --git a/releases/release-7.5.0.md b/releases/release-7.5.0.md index 8cd08d6776f7b..5f397de132afd 100644 --- a/releases/release-7.5.0.md +++ b/releases/release-7.5.0.md @@ -213,7 +213,7 @@ Starting from v7.5.0, the following contents are removed from the `TiDB-communit + TiDB - - Optimize the concurrency model of merging GlobalStats: introduce [`tidb_enable_async_merge_global_stats`](/system-variables.md#tidb_enable_async_merge_global_stats-new-in-v750) to enable simultaneous loading and merging of statistics, which speeds up the generation of GlobalStats on partitioned tables. Optimize the memory usage of merging GlobalStats to avoid OOM and reduce memory allocations. [#47219](https://github.com/pingcap/tidb/issues/47219) @[hawkingrei](https://github.com/hawkingrei) + - Optimize the concurrency model of merging GlobalStats: introduce [`tidb_enable_async_merge_global_stats`](/system-variables.md#tidb_enable_async_merge_global_stats-new-in-v750) to enable simultaneous loading and merging of statistics, which speeds up the generation of global statistics on partitioned tables. Optimize the memory usage of merging global statistics to avoid OOM and reduce memory allocations. [#47219](https://github.com/pingcap/tidb/issues/47219) @[hawkingrei](https://github.com/hawkingrei) - Optimize the `ANALYZE` process: introduce [`tidb_build_sampling_stats_concurrency`](/system-variables.md#tidb_build_sampling_stats_concurrency-new-in-v750) to better control the `ANALYZE` concurrency to reduce resource consumption. Optimize the memory usage of `ANALYZE` to reduce memory allocation and avoid frequent GC by reusing some intermediate results. [#47275](https://github.com/pingcap/tidb/issues/47275) @[hawkingrei](https://github.com/hawkingrei) - Optimize the use of placement policies: support configuring the range of a policy to global and improve the syntax support for common scenarios. [#45384](https://github.com/pingcap/tidb/issues/45384) @[nolouch](https://github.com/nolouch) - Improve the performance of adding indexes with `tidb_ddl_enable_fast_reorg` enabled. In internal tests, v7.5.0 improves the performance by up to 62.5% compared with v6.5.0. [#47757](https://github.com/pingcap/tidb/issues/47757) @[tangenta](https://github.com/tangenta) diff --git a/releases/release-8.1.1.md b/releases/release-8.1.1.md index 76a2d88cea525..ebc61a6f2ff6b 100644 --- a/releases/release-8.1.1.md +++ b/releases/release-8.1.1.md @@ -75,7 +75,7 @@ In v8.1.1, `arbiter` is removed from the `TiDB-community-toolkit` [binary packag - Fix the issue that the `STATE` field in the `INFORMATION_SCHEMA.TIDB_TRX` table is empty due to the `size` of the `STATE` field not being defined [#53026](https://github.com/pingcap/tidb/issues/53026) @[cfzjywxk](https://github.com/cfzjywxk) - Fix the issue that executing the `SELECT DISTINCT CAST(col AS DECIMAL), CAST(col AS SIGNED) FROM ...` query might return incorrect results [#53726](https://github.com/pingcap/tidb/issues/53726) @[hawkingrei](https://github.com/hawkingrei) - Fix the issue that DDL statements incorrectly use etcd and cause tasks to queue up [#52335](https://github.com/pingcap/tidb/issues/52335) @[wjhuang2016](https://github.com/wjhuang2016) - - Fix the issue that the `Distinct_count` information in GlobalStats might be incorrect [#53752](https://github.com/pingcap/tidb/issues/53752) @[hawkingrei](https://github.com/hawkingrei) + - Fix the issue that the `Distinct_count` information in global statistics might be incorrect [#53752](https://github.com/pingcap/tidb/issues/53752) @[hawkingrei](https://github.com/hawkingrei) - Fix the issue that the `tidb_enable_async_merge_global_stats` and `tidb_analyze_partition_concurrency` system variables do not take effect during automatic statistics collection [#53972](https://github.com/pingcap/tidb/issues/53972) @[hi-rustin](https://github.com/hi-rustin) - Fix the issue that the `TIMESTAMPADD()` function goes into an infinite loop when the first argument is `month` and the second argument is negative [#54908](https://github.com/pingcap/tidb/issues/54908) @[xzhangxian1008](https://github.com/xzhangxian1008) - Fix the issue that the Connection Count monitoring metric in Grafana is incorrect when some connections exit before the handshake is complete [#54428](https://github.com/pingcap/tidb/issues/54428) @[YangKeao](https://github.com/YangKeao) diff --git a/releases/release-8.2.0.md b/releases/release-8.2.0.md index 8edc83fea24d9..820dcafb02b80 100644 --- a/releases/release-8.2.0.md +++ b/releases/release-8.2.0.md @@ -302,7 +302,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.2/quick-start-with- - Fix the issue that certain filter conditions in queries might cause the planner module to report an `invalid memory address or nil pointer dereference` error [#53582](https://github.com/pingcap/tidb/issues/53582) [#53580](https://github.com/pingcap/tidb/issues/53580) [#53594](https://github.com/pingcap/tidb/issues/53594) [#53603](https://github.com/pingcap/tidb/issues/53603) @[YangKeao](https://github.com/YangKeao) - Fix the issue that executing `CREATE OR REPLACE VIEW` concurrently might result in the `table doesn't exist` error [#53673](https://github.com/pingcap/tidb/issues/53673) @[tangenta](https://github.com/tangenta) - Fix the issue that the `STATE` field in the `INFORMATION_SCHEMA.TIDB_TRX` table is empty due to the `size` of the `STATE` field not being defined [#53026](https://github.com/pingcap/tidb/issues/53026) @[cfzjywxk](https://github.com/cfzjywxk) - - Fix the issue that the `Distinct_count` information in GlobalStats might be incorrect when `tidb_enable_async_merge_global_stats` is disabled [#53752](https://github.com/pingcap/tidb/issues/53752) @[hawkingrei](https://github.com/hawkingrei) + - Fix the issue that the `Distinct_count` information in global statistics might be incorrect when `tidb_enable_async_merge_global_stats` is disabled [#53752](https://github.com/pingcap/tidb/issues/53752) @[hawkingrei](https://github.com/hawkingrei) - Fix the issue of incorrect WARNINGS information when using Optimizer Hints [#53767](https://github.com/pingcap/tidb/issues/53767) @[hawkingrei](https://github.com/hawkingrei) - Fix the issue that negating a time type results in an incorrect value [#52262](https://github.com/pingcap/tidb/issues/52262) @[solotzg](https://github.com/solotzg) - Fix the issue that `REGEXP()` does not explicitly report an error for empty pattern arguments [#53221](https://github.com/pingcap/tidb/issues/53221) @[yibin87](https://github.com/yibin87) diff --git a/sql-statements/sql-statement-drop-stats.md b/sql-statements/sql-statement-drop-stats.md index 8aeed832f72e6..00e5a5cec68cc 100644 --- a/sql-statements/sql-statement-drop-stats.md +++ b/sql-statements/sql-statement-drop-stats.md @@ -20,7 +20,7 @@ TableName ::= ## Usage -The following statement deletes all statistics of `TableName`. If a partitioned table is specified, this statement deletes statistics of all partitions in this table as well as [GlobalStats generated in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). +The following statement deletes all statistics of `TableName`. If a partitioned table is specified, this statement deletes statistics of all partitions in this table as well as [global statistics generated in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). ```sql DROP STATS TableName @@ -40,7 +40,7 @@ DROP STATS TableName PARTITION PartitionNameList; Query OK, 0 rows affected (0.00 sec) ``` -The following statement only deletes GlobalStats generated in dynamic pruning mode of the specified table. +The following statement only deletes global statistics generated in dynamic pruning mode of the specified table. ```sql DROP STATS TableName GLOBAL; diff --git a/statistics.md b/statistics.md index 16f2bd35e0250..8a9c0bed6b4bf 100644 --- a/statistics.md +++ b/statistics.md @@ -201,19 +201,19 @@ If a table has many columns, collecting statistics on all the columns can cause #### Collect statistics of partitioned tables in dynamic pruning mode -When accessing partitioned tables in [dynamic pruning mode](/partitioned-table.md#dynamic-pruning-mode) (which is the default since v6.3.0), TiDB collects table-level statistics, which is called GlobalStats. Currently, GlobalStats is aggregated from statistics of all partitions. In dynamic pruning mode, a statistics update in any partition of a table can trigger the GlobalStats of that table to be updated. +When accessing partitioned tables in [dynamic pruning mode](/partitioned-table.md#dynamic-pruning-mode) (which is the default since v6.3.0), TiDB collects table-level statistics, meaning global statistics of partitioned tables. Currently, global statistics are aggregated from statistics of all partitions. In dynamic pruning mode, an update to the statistics in any partition of a table can trigger an update to the global statistics for that table. -If statistics of some partitions are empty, or statistics of some columns are missing in some partitions, then the collection behavior is controlled by the [`tidb_skip_missing_partition_stats`](/system-variables.md#tidb_skip_missing_partition_stats-new-in-v730) variable: +If the statistics of some partitions are empty, or statistics of some columns are missing in some partitions, then the collection behavior is controlled by the [`tidb_skip_missing_partition_stats`](/system-variables.md#tidb_skip_missing_partition_stats-new-in-v730) variable: -- When GlobalStats update is triggered and [`tidb_skip_missing_partition_stats`](/system-variables.md#tidb_skip_missing_partition_stats-new-in-v730) is `OFF`: +- When an update to the global statistics is triggered and [`tidb_skip_missing_partition_stats`](/system-variables.md#tidb_skip_missing_partition_stats-new-in-v730) is `OFF`: - - If some partitions have no statistics (such as a new partition that has never been analyzed), GlobalStats generation is interrupted and a warning message is displayed saying that no statistics are available on partitions. + - If some partitions have no statistics (such as a new partition that has never been analyzed), global statistics generation is interrupted and a warning message is displayed saying that no statistics are available on partitions. - - If statistics of some columns are absent in specific partitions (different columns are specified for analyzing in these partitions), GlobalStats generation is interrupted when statistics of these columns are aggregated, and a warning message is displayed saying that statistics of some columns are absent in specific partitions. + - If statistics of some columns are absent in specific partitions (different columns are specified for analyzing in these partitions), global statistics generation is interrupted when statistics of these columns are aggregated, and a warning message is displayed saying that statistics of some columns are absent in specific partitions. -- When GlobalStats update is triggered and [`tidb_skip_missing_partition_stats`](/system-variables.md#tidb_skip_missing_partition_stats-new-in-v730) is `ON`: +- When an update to the global statistics is triggered and [`tidb_skip_missing_partition_stats`](/system-variables.md#tidb_skip_missing_partition_stats-new-in-v730) is `ON`: - - If statistics of all or some columns are missing for some partitions, TiDB skips these missing partition statistics when generating GlobalStats so the generation of GlobalStats is not affected. + - If statistics of all or some columns are missing for some partitions, TiDB skips these missing partition statistics when generating global statistics so the generation of global statistics is not affected. In dynamic pruning mode, the `ANALYZE` configurations of partitions and tables should be the same. Therefore, if you specify the `COLUMNS` configuration following the `ANALYZE TABLE TableName PARTITION PartitionNameList` statement or the `OPTIONS` configuration following `WITH`, TiDB will ignore them and return a warning. diff --git a/system-variables.md b/system-variables.md index 884e991cc31ce..23c16c57270c2 100644 --- a/system-variables.md +++ b/system-variables.md @@ -4801,7 +4801,7 @@ SHOW WARNINGS; - Type: Enumeration - Default value: `dynamic` - Possible values: `static`, `dynamic`, `static-only`, `dynamic-only` -- Specifies whether to use `dynamic` or `static` mode for partitioned tables. Note that dynamic partitioning is effective only after full table-level statistics, or GlobalStats, are collected. Before GlobalStats are collected, TiDB will use the `static` mode instead. For detailed information about GlobalStats, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). For details about the dynamic pruning mode, see [Dynamic Pruning Mode for Partitioned Tables](/partitioned-table.md#dynamic-pruning-mode). +- Specifies whether to use `dynamic` or `static` mode for partitioned tables. Note that dynamic partitioning is effective only after full table-level statistics, or global statistics, are collected. If the `dynamic` mode is selected but the collection of global statistics is not completed, TiDB will use the `static` mode instead. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). For details about the dynamic pruning mode, see [Dynamic Pruning Mode for Partitioned Tables](/partitioned-table.md#dynamic-pruning-mode). ### tidb_persist_analyze_options New in v5.4.0 @@ -5320,10 +5320,10 @@ Query OK, 0 rows affected, 1 warning (0.00 sec) - Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No - Type: Boolean - Default value: `ON` -- When accessing a partitioned table in [dynamic pruning mode](/partitioned-table.md#dynamic-pruning-mode), TiDB aggregates the statistics of each partition to generate GlobalStats. This variable controls the generation of GlobalStats when partition statistics are missing. +- When accessing a partitioned table in [dynamic pruning mode](/partitioned-table.md#dynamic-pruning-mode), TiDB aggregates the statistics of each partition to generate global statistics. This variable controls the generation of global statistics when partition statistics are missing. - - If this variable is `ON`, TiDB skips missing partition statistics when generating GlobalStats so the generation of GlobalStats is not affected. - - If this variable is `OFF`, TiDB stops generating GlobalStats when it detects any missing partition statistics. + - If this variable is `ON`, TiDB skips missing partition statistics when generating global statistics so the generation of global statistics is not affected. + - If this variable is `OFF`, TiDB stops generating global statistics when it detects any missing partition statistics. ### tidb_skip_utf8_check diff --git a/tiflash-upgrade-guide.md b/tiflash-upgrade-guide.md index f36051c145166..4308338f57864 100644 --- a/tiflash-upgrade-guide.md +++ b/tiflash-upgrade-guide.md @@ -79,7 +79,7 @@ If you do not enable [dynamic pruning mode](/partitioned-table.md#dynamic-prunin - TiDB v6.0 and earlier: Dynamic pruning is disabled by default. The setting of dynamic pruning after an upgrade inherits that of the previous version. That is, dynamic pruning will not be enabled (or disabled) automatically after an upgrade. - After an upgrade, to enable dynamic pruning, set `tidb_partition_prune_mode` to `dynamic` and manually update GlobalStats of partitioned tables. For details, see [Dynamic pruning mode](/partitioned-table.md#dynamic-pruning-mode). + After an upgrade, to enable dynamic pruning, set `tidb_partition_prune_mode` to `dynamic` and manually update global statistics of partitioned tables. For details, see [Dynamic pruning mode](/partitioned-table.md#dynamic-pruning-mode). ## From v5.x or v6.0 to v6.2 From 191ee338b1166b5b9ce3bb780172f67247c0617b Mon Sep 17 00:00:00 2001 From: lilin90 Date: Thu, 14 Nov 2024 18:24:08 +0800 Subject: [PATCH 04/11] Refine wording --- releases/release-6.3.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.3.0.md b/releases/release-6.3.0.md index 85b15fe0c4790..7e1c0b6c321ff 100644 --- a/releases/release-6.3.0.md +++ b/releases/release-6.3.0.md @@ -147,7 +147,7 @@ In v6.3.0-DMR, the key new features and improvements are as follows: * Use `static` partition pruning when global statistics are not ready [#37535](https://github.com/pingcap/tidb/issues/37535) @[Yisaer](https://github.com/Yisaer) - When [`dynamic pruning`](/partitioned-table.md#dynamic-pruning-mode) is enabled, the optimizer selects execution plans based on [GlobalStats](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). Before global statistics are fully collected, using pseudo statistics might cause performance regression. In v6.3.0, this issue is addressed by maintaining the `static` mode if you enable dynamic pruning before global statistics are collected. TiDB remains in the `static` mode until global statistics are collected. This ensures performance stability when you change the partition pruning settings. + When [`dynamic pruning`](/partitioned-table.md#dynamic-pruning-mode) is enabled, the optimizer selects execution plans based on [global statistics](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). Before global statistics are fully collected, using pseudo statistics might cause performance regression. In v6.3.0, if you enable the dynamic pruning mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. This ensures performance stability when you change the partition pruning settings. ### Ease of use From 87f8bdd14d6dcc9855fe57d0ae7783c28cd7e785 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Thu, 14 Nov 2024 18:24:53 +0800 Subject: [PATCH 05/11] Update release-6.3.0.md --- releases/release-6.3.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.3.0.md b/releases/release-6.3.0.md index 7e1c0b6c321ff..09b2fbf5a73a5 100644 --- a/releases/release-6.3.0.md +++ b/releases/release-6.3.0.md @@ -147,7 +147,7 @@ In v6.3.0-DMR, the key new features and improvements are as follows: * Use `static` partition pruning when global statistics are not ready [#37535](https://github.com/pingcap/tidb/issues/37535) @[Yisaer](https://github.com/Yisaer) - When [`dynamic pruning`](/partitioned-table.md#dynamic-pruning-mode) is enabled, the optimizer selects execution plans based on [global statistics](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). Before global statistics are fully collected, using pseudo statistics might cause performance regression. In v6.3.0, if you enable the dynamic pruning mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. This ensures performance stability when you change the partition pruning settings. + When [`dynamic pruning`](/partitioned-table.md#dynamic-pruning-mode) is enabled, the optimizer selects execution plans based on [global statistics](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). Before global statistics are fully collected, using pseudo statistics might cause performance regression. In v6.3.0, if you enable the dynamic pruning mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. This ensures performance stability when you change the partition pruning settings. ### Ease of use From c55f18a25d923c0eb0da93f671788352aaab8dba Mon Sep 17 00:00:00 2001 From: lilin90 Date: Fri, 15 Nov 2024 14:30:17 +0800 Subject: [PATCH 06/11] Update descriptions for consistency --- partitioned-table.md | 2 +- system-variables.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/partitioned-table.md b/partitioned-table.md index 42da76865b252..dfba3f6437219 100644 --- a/partitioned-table.md +++ b/partitioned-table.md @@ -1926,7 +1926,7 @@ This variable is only used in table creation. After the table is created, modify ### Dynamic pruning mode -TiDB accesses partitioned tables in either `dynamic` or `static` mode. `dynamic` mode is used by default since v6.3.0. However, dynamic partitioning is effective only after the full table-level statistics, or global statistics, are collected. If the `dynamic` mode is selected but the collection of global statistics is not completed, TiDB will use the `static` mode instead. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). +TiDB accesses partitioned tables in either `dynamic` or `static` mode. `dynamic` mode is used by default since v6.3.0. However, dynamic partitioning is effective only after the full table-level statistics, or global statistics, are collected. If you enable the `dynamic` mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). {{< copyable "sql" >}} diff --git a/system-variables.md b/system-variables.md index 23c16c57270c2..792062cab710f 100644 --- a/system-variables.md +++ b/system-variables.md @@ -4801,7 +4801,7 @@ SHOW WARNINGS; - Type: Enumeration - Default value: `dynamic` - Possible values: `static`, `dynamic`, `static-only`, `dynamic-only` -- Specifies whether to use `dynamic` or `static` mode for partitioned tables. Note that dynamic partitioning is effective only after full table-level statistics, or global statistics, are collected. If the `dynamic` mode is selected but the collection of global statistics is not completed, TiDB will use the `static` mode instead. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). For details about the dynamic pruning mode, see [Dynamic Pruning Mode for Partitioned Tables](/partitioned-table.md#dynamic-pruning-mode). +- Specifies whether to use `dynamic` or `static` mode for partitioned tables. Note that dynamic partitioning is effective only after full table-level statistics, or global statistics, are collected. If you enable the `dynamic` mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). For details about the dynamic pruning mode, see [Dynamic Pruning Mode for Partitioned Tables](/partitioned-table.md#dynamic-pruning-mode). ### tidb_persist_analyze_options New in v5.4.0 From f1e87ea5251e4963a0c39670dfecbbc91d352053 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Fri, 15 Nov 2024 14:31:51 +0800 Subject: [PATCH 07/11] Update format --- releases/release-6.3.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-6.3.0.md b/releases/release-6.3.0.md index 09b2fbf5a73a5..dfd00a9dd92a1 100644 --- a/releases/release-6.3.0.md +++ b/releases/release-6.3.0.md @@ -147,7 +147,7 @@ In v6.3.0-DMR, the key new features and improvements are as follows: * Use `static` partition pruning when global statistics are not ready [#37535](https://github.com/pingcap/tidb/issues/37535) @[Yisaer](https://github.com/Yisaer) - When [`dynamic pruning`](/partitioned-table.md#dynamic-pruning-mode) is enabled, the optimizer selects execution plans based on [global statistics](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). Before global statistics are fully collected, using pseudo statistics might cause performance regression. In v6.3.0, if you enable the dynamic pruning mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. This ensures performance stability when you change the partition pruning settings. + When [`dynamic pruning`](/partitioned-table.md#dynamic-pruning-mode) is enabled, the optimizer selects execution plans based on [global statistics](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). Before global statistics are fully collected, using pseudo statistics might cause performance regression. In v6.3.0, if you enable the `dynamic` pruning mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. This ensures performance stability when you change the partition pruning settings. ### Ease of use From e18f3e155231f9b493cf6a0b4049bdc44c03d6bb Mon Sep 17 00:00:00 2001 From: lilin90 Date: Fri, 15 Nov 2024 14:35:19 +0800 Subject: [PATCH 08/11] Make wording consistent --- optimizer-fix-controls.md | 2 +- partitioned-table.md | 2 +- releases/release-7.5.0.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/optimizer-fix-controls.md b/optimizer-fix-controls.md index 0fc68bc3fce56..24dabc5450899 100644 --- a/optimizer-fix-controls.md +++ b/optimizer-fix-controls.md @@ -36,7 +36,7 @@ SET SESSION tidb_opt_fix_control = '44262:ON,44389:ON'; - Default value: `OFF` - Possible values: `ON`, `OFF` -- This variable controls whether to allow the use of [Dynamic pruning mode](/partitioned-table.md#dynamic-pruning-mode) to access the partitioned table when the [GlobalStats](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode) are missing. +- This variable controls whether to allow the use of [Dynamic pruning mode](/partitioned-table.md#dynamic-pruning-mode) to access the partitioned table when the [global statistics](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode) are missing. ### [`44389`](https://github.com/pingcap/tidb/issues/44389) New in v6.5.3 and v7.2.0 diff --git a/partitioned-table.md b/partitioned-table.md index dfba3f6437219..64f3d32db67ce 100644 --- a/partitioned-table.md +++ b/partitioned-table.md @@ -1958,7 +1958,7 @@ show stats_meta where table_name like "t"; 3 rows in set (0.01 sec) ``` -To make sure that the statistics used by SQL statements are correct after you enable global `dynamic` pruning mode, you need to manually trigger `analyze` on the tables or on a partition of the table to obtain GlobalStats. +To make sure that the statistics used by SQL statements are correct after you enable global `dynamic` pruning mode, you need to manually trigger `analyze` on the tables or on a partition of the table to obtain global statistics. {{< copyable "sql" >}} diff --git a/releases/release-7.5.0.md b/releases/release-7.5.0.md index 5f397de132afd..464f92595996d 100644 --- a/releases/release-7.5.0.md +++ b/releases/release-7.5.0.md @@ -213,7 +213,7 @@ Starting from v7.5.0, the following contents are removed from the `TiDB-communit + TiDB - - Optimize the concurrency model of merging GlobalStats: introduce [`tidb_enable_async_merge_global_stats`](/system-variables.md#tidb_enable_async_merge_global_stats-new-in-v750) to enable simultaneous loading and merging of statistics, which speeds up the generation of global statistics on partitioned tables. Optimize the memory usage of merging global statistics to avoid OOM and reduce memory allocations. [#47219](https://github.com/pingcap/tidb/issues/47219) @[hawkingrei](https://github.com/hawkingrei) + - Optimize the concurrency model of merging global statistics: introduce [`tidb_enable_async_merge_global_stats`](/system-variables.md#tidb_enable_async_merge_global_stats-new-in-v750) to enable simultaneous loading and merging of statistics, which speeds up the generation of global statistics on partitioned tables. Optimize the memory usage of merging global statistics to avoid OOM and reduce memory allocations. [#47219](https://github.com/pingcap/tidb/issues/47219) @[hawkingrei](https://github.com/hawkingrei) - Optimize the `ANALYZE` process: introduce [`tidb_build_sampling_stats_concurrency`](/system-variables.md#tidb_build_sampling_stats_concurrency-new-in-v750) to better control the `ANALYZE` concurrency to reduce resource consumption. Optimize the memory usage of `ANALYZE` to reduce memory allocation and avoid frequent GC by reusing some intermediate results. [#47275](https://github.com/pingcap/tidb/issues/47275) @[hawkingrei](https://github.com/hawkingrei) - Optimize the use of placement policies: support configuring the range of a policy to global and improve the syntax support for common scenarios. [#45384](https://github.com/pingcap/tidb/issues/45384) @[nolouch](https://github.com/nolouch) - Improve the performance of adding indexes with `tidb_ddl_enable_fast_reorg` enabled. In internal tests, v7.5.0 improves the performance by up to 62.5% compared with v6.5.0. [#47757](https://github.com/pingcap/tidb/issues/47757) @[tangenta](https://github.com/tangenta) From 5b026ecb480dc77ba6a7a255c95ce7e7032c8df4 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Fri, 15 Nov 2024 14:36:57 +0800 Subject: [PATCH 09/11] Refine wording --- partitioned-table.md | 2 +- system-variables.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/partitioned-table.md b/partitioned-table.md index 64f3d32db67ce..5dc0054e1aa3b 100644 --- a/partitioned-table.md +++ b/partitioned-table.md @@ -1926,7 +1926,7 @@ This variable is only used in table creation. After the table is created, modify ### Dynamic pruning mode -TiDB accesses partitioned tables in either `dynamic` or `static` mode. `dynamic` mode is used by default since v6.3.0. However, dynamic partitioning is effective only after the full table-level statistics, or global statistics, are collected. If you enable the `dynamic` mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). +TiDB accesses partitioned tables in either `dynamic` or `static` mode. `dynamic` mode is used by default since v6.3.0. However, dynamic partitioning is effective only after the full table-level statistics, or global statistics, are collected. If you enable the `dynamic` pruning mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). {{< copyable "sql" >}} diff --git a/system-variables.md b/system-variables.md index 792062cab710f..be4863081f010 100644 --- a/system-variables.md +++ b/system-variables.md @@ -4801,7 +4801,7 @@ SHOW WARNINGS; - Type: Enumeration - Default value: `dynamic` - Possible values: `static`, `dynamic`, `static-only`, `dynamic-only` -- Specifies whether to use `dynamic` or `static` mode for partitioned tables. Note that dynamic partitioning is effective only after full table-level statistics, or global statistics, are collected. If you enable the `dynamic` mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). For details about the dynamic pruning mode, see [Dynamic Pruning Mode for Partitioned Tables](/partitioned-table.md#dynamic-pruning-mode). +- Specifies whether to use `dynamic` or `static` mode for partitioned tables. Note that dynamic partitioning is effective only after full table-level statistics, or global statistics, are collected. If you enable the `dynamic` pruning mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). For details about the dynamic pruning mode, see [Dynamic Pruning Mode for Partitioned Tables](/partitioned-table.md#dynamic-pruning-mode). ### tidb_persist_analyze_options New in v5.4.0 From 4e719e50d22c8391a8993a90cfc3cb74a2708451 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Fri, 15 Nov 2024 17:03:46 +0800 Subject: [PATCH 10/11] Apply suggestions from code review Co-authored-by: Grace Cai --- optimizer-fix-controls.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimizer-fix-controls.md b/optimizer-fix-controls.md index 9b8501fde1c64..83a31304307d9 100644 --- a/optimizer-fix-controls.md +++ b/optimizer-fix-controls.md @@ -36,7 +36,7 @@ SET SESSION tidb_opt_fix_control = '44262:ON,44389:ON'; - Default value: `OFF` - Possible values: `ON`, `OFF` -- This variable controls whether to allow the use of [Dynamic pruning mode](/partitioned-table.md#dynamic-pruning-mode) to access the partitioned table when the [global statistics](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode) are missing. +- This variable controls whether to allow the use of [Dynamic pruning mode](/partitioned-table.md#dynamic-pruning-mode) to access a partitioned table when the [global statistics](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode) of that table are missing. ### [`44389`](https://github.com/pingcap/tidb/issues/44389) New in v6.5.3 and v7.2.0 From b1b9414face9f5d38764eb876df634171d5a7f56 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Mon, 18 Nov 2024 10:25:16 +0800 Subject: [PATCH 11/11] Apply suggestions from code review Co-authored-by: Grace Cai --- partitioned-table.md | 2 +- releases/release-6.3.0.md | 2 +- system-variables.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/partitioned-table.md b/partitioned-table.md index 30b61acc95d01..2149ed063d0d5 100644 --- a/partitioned-table.md +++ b/partitioned-table.md @@ -1911,7 +1911,7 @@ select * from t; ### Dynamic pruning mode -TiDB accesses partitioned tables in either `dynamic` or `static` mode. `dynamic` mode is used by default since v6.3.0. However, dynamic partitioning is effective only after the full table-level statistics, or global statistics, are collected. If you enable the `dynamic` pruning mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). +TiDB accesses partitioned tables in either `dynamic` or `static` mode. `dynamic` mode is used by default since v6.3.0. However, dynamic partitioning is effective only after the full table-level statistics, or global statistics, are collected. If you enable the `dynamic` pruning mode before global statistics collection is completed, TiDB remains in the `static` mode until global statistics are fully collected. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). {{< copyable "sql" >}} diff --git a/releases/release-6.3.0.md b/releases/release-6.3.0.md index dfd00a9dd92a1..84b106fbfafa1 100644 --- a/releases/release-6.3.0.md +++ b/releases/release-6.3.0.md @@ -147,7 +147,7 @@ In v6.3.0-DMR, the key new features and improvements are as follows: * Use `static` partition pruning when global statistics are not ready [#37535](https://github.com/pingcap/tidb/issues/37535) @[Yisaer](https://github.com/Yisaer) - When [`dynamic pruning`](/partitioned-table.md#dynamic-pruning-mode) is enabled, the optimizer selects execution plans based on [global statistics](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). Before global statistics are fully collected, using pseudo statistics might cause performance regression. In v6.3.0, if you enable the `dynamic` pruning mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. This ensures performance stability when you change the partition pruning settings. + When [`dynamic pruning`](/partitioned-table.md#dynamic-pruning-mode) is enabled, the optimizer selects execution plans based on [global statistics](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). Before global statistics are fully collected, using pseudo statistics might cause performance regression. In v6.3.0, if you enable the `dynamic` pruning mode before global statistics collection is completed, TiDB remains in the `static` mode until global statistics are fully collected. This ensures performance stability when you change the partition pruning settings. ### Ease of use diff --git a/system-variables.md b/system-variables.md index 833238a00dbc6..e7a9e50d69f54 100644 --- a/system-variables.md +++ b/system-variables.md @@ -4867,7 +4867,7 @@ SHOW WARNINGS; - Type: Enumeration - Default value: `dynamic` - Possible values: `static`, `dynamic`, `static-only`, `dynamic-only` -- Specifies whether to use `dynamic` or `static` mode for partitioned tables. Note that dynamic partitioning is effective only after full table-level statistics, or global statistics, are collected. If you enable the `dynamic` pruning mode before global statistics collection is completed, TiDB will remain in the `static` mode until global statistics are fully collected. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). For details about the dynamic pruning mode, see [Dynamic Pruning Mode for Partitioned Tables](/partitioned-table.md#dynamic-pruning-mode). +- Specifies whether to use `dynamic` or `static` mode for partitioned tables. Note that dynamic partitioning is effective only after full table-level statistics, or global statistics, are collected. If you enable the `dynamic` pruning mode before global statistics collection is completed, TiDB remains in the `static` mode until global statistics are fully collected. For detailed information about global statistics, see [Collect statistics of partitioned tables in dynamic pruning mode](/statistics.md#collect-statistics-of-partitioned-tables-in-dynamic-pruning-mode). For details about the dynamic pruning mode, see [Dynamic Pruning Mode for Partitioned Tables](/partitioned-table.md#dynamic-pruning-mode). ### tidb_persist_analyze_options New in v5.4.0