From 4885a08c752fb1dcae19cef8523082e4820486c7 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Mon, 16 Aug 2021 15:23:35 +0800 Subject: [PATCH 1/5] Add information of hashAgg for memory control --- configure-memory-usage.md | 80 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/configure-memory-usage.md b/configure-memory-usage.md index 414b22aa92ba9..e65c6e549643b 100644 --- a/configure-memory-usage.md +++ b/configure-memory-usage.md @@ -118,3 +118,83 @@ The following example constructs a memory-intensive SQL statement that triggers * `record path` indicates the directory of status files. 5. You can see a set of files in the directory of status files (In the above example, the directory is `/tmp/1000_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage/record`), including `goroutinue`, `heap`, and `running_sql`. These three files are suffixed with the time when status files are logged. They respectively record goroutine stack information, the usage status of heap memory, and the running SQL information when the alarm is triggered. For the format of log content in `running_sql`, refer to [`expensive-queries`](/identify-expensive-queries.md). + +## Other tidb-server memory control behaviors + +### Flow control + +- TiDB supports dynamic memory control feature for the operator that reads data. By default, this operator enables the maximum number of threads that [`tidb_disql_scan_concurrency`](/system-variables.md#tidb_distsql_scan_concurrency) allows to read data. When the memory usage of a single SQL statement exceeds [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) each time, the operator that reads data stops one thread. + +- This flow control behavior is controlled by the system variable [`tidb_enable_rate_limit_action`](/system-variables.md#tidb_enable_rate_limit_action). +- When the flow control behavior is triggered, TiDB outputs a log containing the key word `memory exceeds quota, destroy one token now`. + +### Disk spill + +TiDB supports disk spill feature for execution operators. When the memory usage of a SQL statement exceeds the memory quota, tidb-server can spill the intermediate data of execution operators to the disk to relieve memory pressure. Operators supporting disk spill include Sort, MergeJoin, HashJoin, and HashAgg. + +- The disk spill action is jointly controlled by parameters [`mem-quota-query`](/tidb-configuration-file.md#mem-quota-query), [`oom-use-tmp-storage`](/tidb-configuration-file.md#oom-use-tmp-storage), [`tmp-storage-path`](/tidb-configuration-file.md#tmp-storage-path), and [`tmp-storage-quota`](/tidb-configuration-file.md#tmp-storage-quota). +- When the disk spill is triggered, TiDB outputs a log containing the key word `memory exceeds quota, spill to disk now` or `memory exceeds quota, set aggregate mode to spill-mode`. +- Disk spill for operators Sort, MergeJoin, and HashJoin is new in v4.0.0; disk spill for operator HashAgg is new in v5.2.0. +- When SQL statements containing Sort, MergeJoin, or HashJoin cause OOM, TiDB triggers disk spill by default. When SQL statements containing HashAgg cause OOM, TiDB does not trigger disk spill by default. You can configure the system variable `tidb_executor_concurrency = 1` to trigger disk spill for HashAgg. + +> **Note:** +> +> + Currently, it is not supported to use the aggregate functions with `DISTINCT` option to spill the disk for HashAgg. When you use the aggregate functions with `DISTINCT` option with too much memory, the disk spill fails. +The following example creates a SQL statement occupying too much memory to display the disk spill feature for HashAgg: + +1. Configure the memory quota of a SQL statement to 1GB (1 GB by default)： + + {{< copyable "sql" >}} + + ```sql + set tidb_mem_quota_query = 1 << 30; + ``` + +2. Create a single table `CREATE TABLE t(a int);` and insert 256 rows of different data. + +3. Execute the following SQL statement: + + {{< copyable "sql" >}} + + ```sql + [tidb]> explain analyze select /*+ HASH_AGG() */ count(*) from t t1 join t t2 join t t3 group by t1.a, t2.a, t3.a; + ``` + + Because this SQL statement occupies too much memory, the following error message "out of memory quota" is returned: + + ```sql + ERROR 1105 (HY000): Out Of Memory Quota![conn_id=3] + ``` + +4. Configure the system variable `tidb_executor_concurrency` to 1. With this configuration, when out of memory, HashAgg automatically tries to trigger disk spill. + + {{< copyable "sql" >}} + + ```sql + set tidb_executor_concurrency = 1; + ``` + +5. Execute the same SQL statement. You can find this time the statement can be successfully executed and no error message is returned. From the following detailed execution plan, you can see that HashAgg used 600MB of hard disk space. + + {{< copyable "sql" >}} + + ```sql + [tidb]> explain analyze select /*+ HASH_AGG() */ count(*) from t t1 join t t2 join t t3 group by t1.a, t2.a, t3.a; + ``` + + ```sql + +---------------------------------+-------------+----------+-----------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------+-----------+----------+ + | id | estRows | actRows | task | access object | execution info | operator info | memory | disk | + +---------------------------------+-------------+----------+-----------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------+-----------+----------+ + | HashAgg_11 | 204.80 | 16777216 | root | | time:1m37.4s, loops:16385 | group by:test.t.a, test.t.a, test.t.a, funcs:count(1)->Column#7 | 1.13 GB | 600.0 MB | + | └─HashJoin_12 | 16777216.00 | 16777216 | root | | time:21.5s, loops:16385, build_hash_table:{total:267.2µs, fetch:228.9µs, build:38.2µs}, probe:{concurrency:1, total:35s, max:35s, probe:35s, fetch:962.2µs} | CARTESIAN inner join | 8.23 KB | 4 KB | + | ├─TableReader_21(Build) | 256.00 | 256 | root | | time:87.2µs, loops:2, cop_task: {num: 1, max: 150µs, proc_keys: 0, rpc_num: 1, rpc_time: 145.1µs, copr_cache_hit_ratio: 0.00} | data:TableFullScan_20 | 885 Bytes | N/A | + | │ └─TableFullScan_20 | 256.00 | 256 | cop[tikv] | table:t3 | tikv_task:{time:23.2µs, loops:256} | keep order:false, stats:pseudo | N/A | N/A | + | └─HashJoin_14(Probe) | 65536.00 | 65536 | root | | time:728.1µs, loops:65, build_hash_table:{total:307.5µs, fetch:277.6µs, build:29.9µs}, probe:{concurrency:1, total:34.3s, max:34.3s, probe:34.3s, fetch:278µs} | CARTESIAN inner join | 8.23 KB | 4 KB | + | ├─TableReader_19(Build) | 256.00 | 256 | root | | time:126.2µs, loops:2, cop_task: {num: 1, max: 308.4µs, proc_keys: 0, rpc_num: 1, rpc_time: 295.3µs, copr_cache_hit_ratio: 0.00} | data:TableFullScan_18 | 885 Bytes | N/A | + | │ └─TableFullScan_18 | 256.00 | 256 | cop[tikv] | table:t2 | tikv_task:{time:79.2µs, loops:256} | keep order:false, stats:pseudo | N/A | N/A | + | └─TableReader_17(Probe) | 256.00 | 256 | root | | time:211.1µs, loops:2, cop_task: {num: 1, max: 295.5µs, proc_keys: 0, rpc_num: 1, rpc_time: 279.7µs, copr_cache_hit_ratio: 0.00} | data:TableFullScan_16 | 885 Bytes | N/A | + | └─TableFullScan_16 | 256.00 | 256 | cop[tikv] | table:t1 | tikv_task:{time:71.4µs, loops:256} | keep order:false, stats:pseudo | N/A | N/A | + +---------------------------------+-------------+----------+-----------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------+-----------+----------+ + 9 rows in set (1 min 37.428 sec) + ``` \ No newline at end of file From dc01cd3f5292fc4690b62ecc708d980aac3a088e Mon Sep 17 00:00:00 2001 From: Liuxiaozhen12 <82579298+Liuxiaozhen12@users.noreply.github.com> Date: Thu, 19 Aug 2021 12:39:31 +0800 Subject: [PATCH 2/5] Apply suggestions from code review Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- configure-memory-usage.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/configure-memory-usage.md b/configure-memory-usage.md index e65c6e549643b..a9770357bccc6 100644 --- a/configure-memory-usage.md +++ b/configure-memory-usage.md @@ -119,30 +119,31 @@ The following example constructs a memory-intensive SQL statement that triggers 5. You can see a set of files in the directory of status files (In the above example, the directory is `/tmp/1000_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage/record`), including `goroutinue`, `heap`, and `running_sql`. These three files are suffixed with the time when status files are logged. They respectively record goroutine stack information, the usage status of heap memory, and the running SQL information when the alarm is triggered. For the format of log content in `running_sql`, refer to [`expensive-queries`](/identify-expensive-queries.md). -## Other tidb-server memory control behaviors +## Other memory control behaviors of tidb-server ### Flow control -- TiDB supports dynamic memory control feature for the operator that reads data. By default, this operator enables the maximum number of threads that [`tidb_disql_scan_concurrency`](/system-variables.md#tidb_distsql_scan_concurrency) allows to read data. When the memory usage of a single SQL statement exceeds [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) each time, the operator that reads data stops one thread. +- TiDB supports dynamic memory control for the operator that reads data. By default, this operator uses the maximum number of threads that [`tidb_disql_scan_concurrency`](/system-variables.md#tidb_distsql_scan_concurrency) allows to read data. When the memory usage of a single SQL execution exceeds [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) each time, the operator that reads data stops one thread. - This flow control behavior is controlled by the system variable [`tidb_enable_rate_limit_action`](/system-variables.md#tidb_enable_rate_limit_action). -- When the flow control behavior is triggered, TiDB outputs a log containing the key word `memory exceeds quota, destroy one token now`. +- When the flow control behavior is triggered, TiDB outputs a log containing the key words `memory exceeds quota, destroy one token now`. ### Disk spill -TiDB supports disk spill feature for execution operators. When the memory usage of a SQL statement exceeds the memory quota, tidb-server can spill the intermediate data of execution operators to the disk to relieve memory pressure. Operators supporting disk spill include Sort, MergeJoin, HashJoin, and HashAgg. +TiDB supports disk spill for execution operators. When the memory usage of a SQL execution exceeds the memory quota, tidb-server can spill the intermediate data of execution operators to the disk to relieve memory pressure. Operators supporting disk spill include Sort, MergeJoin, HashJoin, and HashAgg. -- The disk spill action is jointly controlled by parameters [`mem-quota-query`](/tidb-configuration-file.md#mem-quota-query), [`oom-use-tmp-storage`](/tidb-configuration-file.md#oom-use-tmp-storage), [`tmp-storage-path`](/tidb-configuration-file.md#tmp-storage-path), and [`tmp-storage-quota`](/tidb-configuration-file.md#tmp-storage-quota). +- The disk spill behavior is jointly controlled by the [`mem-quota-query`](/tidb-configuration-file.md#mem-quota-query), [`oom-use-tmp-storage`](/tidb-configuration-file.md#oom-use-tmp-storage), [`tmp-storage-path`](/tidb-configuration-file.md#tmp-storage-path), and [`tmp-storage-quota`](/tidb-configuration-file.md#tmp-storage-quota) parameters. - When the disk spill is triggered, TiDB outputs a log containing the key word `memory exceeds quota, spill to disk now` or `memory exceeds quota, set aggregate mode to spill-mode`. -- Disk spill for operators Sort, MergeJoin, and HashJoin is new in v4.0.0; disk spill for operator HashAgg is new in v5.2.0. -- When SQL statements containing Sort, MergeJoin, or HashJoin cause OOM, TiDB triggers disk spill by default. When SQL statements containing HashAgg cause OOM, TiDB does not trigger disk spill by default. You can configure the system variable `tidb_executor_concurrency = 1` to trigger disk spill for HashAgg. +- Disk spill for the Sort, MergeJoin, and HashJoin operator is introduced in v4.0.0; disk spill for the HashAgg operator is introduced in v5.2.0. +- When the SQL executions containing Sort, MergeJoin, or HashJoin cause OOM, TiDB triggers disk spill by default. When SQL executions containing HashAgg cause OOM, TiDB does not trigger disk spill by default. You can configure the system variable `tidb_executor_concurrency = 1` to trigger disk spill for HashAgg. > **Note:** > -> + Currently, it is not supported to use the aggregate functions with `DISTINCT` option to spill the disk for HashAgg. When you use the aggregate functions with `DISTINCT` option with too much memory, the disk spill fails. -The following example creates a SQL statement occupying too much memory to display the disk spill feature for HashAgg: +> The disk spill for HashAgg does not support SQL executions containing the `DISTINCT` aggregate function. When a SQL execution containing a `DISTINCT` aggregate function uses too much memory, the disk spill does not apply. -1. Configure the memory quota of a SQL statement to 1GB (1 GB by default)： +The following example uses a memory-consuming SQL statement to demonstrate the disk spill feature for HashAgg: + +1. Configure the memory quota of a SQL statement to 1GB (1 GB by default): {{< copyable "sql" >}} @@ -160,7 +161,7 @@ The following example creates a SQL statement occupying too much memory to displ [tidb]> explain analyze select /*+ HASH_AGG() */ count(*) from t t1 join t t2 join t t3 group by t1.a, t2.a, t3.a; ``` - Because this SQL statement occupies too much memory, the following error message "out of memory quota" is returned: + Because executing this SQL statement occupies too much memory, the following "Out of Memory Quota" error message is returned: ```sql ERROR 1105 (HY000): Out Of Memory Quota![conn_id=3] @@ -174,7 +175,7 @@ The following example creates a SQL statement occupying too much memory to displ set tidb_executor_concurrency = 1; ``` -5. Execute the same SQL statement. You can find this time the statement can be successfully executed and no error message is returned. From the following detailed execution plan, you can see that HashAgg used 600MB of hard disk space. +5. Execute the same SQL statement. You can find that this time, the statement is successfully executed and no error message is returned. From the following detailed execution plan, you can see that HashAgg has used 600 MB of hard disk space. {{< copyable "sql" >}} From 00afc5fe45386be712e007690e42d4a00be58438 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 19 Aug 2021 12:42:15 +0800 Subject: [PATCH 3/5] modify key word to keyword --- configure-memory-usage.md | 4 ++-- troubleshoot-write-conflicts.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/configure-memory-usage.md b/configure-memory-usage.md index e65c6e549643b..8f5dd1260f098 100644 --- a/configure-memory-usage.md +++ b/configure-memory-usage.md @@ -126,14 +126,14 @@ The following example constructs a memory-intensive SQL statement that triggers - TiDB supports dynamic memory control feature for the operator that reads data. By default, this operator enables the maximum number of threads that [`tidb_disql_scan_concurrency`](/system-variables.md#tidb_distsql_scan_concurrency) allows to read data. When the memory usage of a single SQL statement exceeds [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) each time, the operator that reads data stops one thread. - This flow control behavior is controlled by the system variable [`tidb_enable_rate_limit_action`](/system-variables.md#tidb_enable_rate_limit_action). -- When the flow control behavior is triggered, TiDB outputs a log containing the key word `memory exceeds quota, destroy one token now`. +- When the flow control behavior is triggered, TiDB outputs a log containing the keywords `memory exceeds quota, destroy one token now`. ### Disk spill TiDB supports disk spill feature for execution operators. When the memory usage of a SQL statement exceeds the memory quota, tidb-server can spill the intermediate data of execution operators to the disk to relieve memory pressure. Operators supporting disk spill include Sort, MergeJoin, HashJoin, and HashAgg. - The disk spill action is jointly controlled by parameters [`mem-quota-query`](/tidb-configuration-file.md#mem-quota-query), [`oom-use-tmp-storage`](/tidb-configuration-file.md#oom-use-tmp-storage), [`tmp-storage-path`](/tidb-configuration-file.md#tmp-storage-path), and [`tmp-storage-quota`](/tidb-configuration-file.md#tmp-storage-quota). -- When the disk spill is triggered, TiDB outputs a log containing the key word `memory exceeds quota, spill to disk now` or `memory exceeds quota, set aggregate mode to spill-mode`. +- When the disk spill is triggered, TiDB outputs a log containing the keywords `memory exceeds quota, spill to disk now` or `memory exceeds quota, set aggregate mode to spill-mode`. - Disk spill for operators Sort, MergeJoin, and HashJoin is new in v4.0.0; disk spill for operator HashAgg is new in v5.2.0. - When SQL statements containing Sort, MergeJoin, or HashJoin cause OOM, TiDB triggers disk spill by default. When SQL statements containing HashAgg cause OOM, TiDB does not trigger disk spill by default. You can configure the system variable `tidb_executor_concurrency = 1` to trigger disk spill for HashAgg. diff --git a/troubleshoot-write-conflicts.md b/troubleshoot-write-conflicts.md index 97bcb5a900958..8a8feb7754f36 100644 --- a/troubleshoot-write-conflicts.md +++ b/troubleshoot-write-conflicts.md @@ -46,7 +46,7 @@ In the TiDB Grafana panel, check the following monitoring metrics under **KV Err ![kv-retry-duration](/media/troubleshooting-write-conflict-kv-retry-duration.png) -You can also use `[kv:9007]Write conflict` as the key word to search in the TiDB log. The key word also indicates the write conflict exists in the cluster. +You can also use `[kv:9007]Write conflict` as the keywords to search in the TiDB log. The keywords also indicates the write conflict exists in the cluster. ## Resolve write conflicts From 687efafcee019bffd69c5aff4d82d10de0f4f33c Mon Sep 17 00:00:00 2001 From: Liuxiaozhen12 <82579298+Liuxiaozhen12@users.noreply.github.com> Date: Thu, 19 Aug 2021 15:03:31 +0800 Subject: [PATCH 4/5] Apply suggestions from code review Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- configure-memory-usage.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/configure-memory-usage.md b/configure-memory-usage.md index 0edfd62c1f8b7..c754c221b270b 100644 --- a/configure-memory-usage.md +++ b/configure-memory-usage.md @@ -132,10 +132,10 @@ The following example constructs a memory-intensive SQL statement that triggers TiDB supports disk spill for execution operators. When the memory usage of a SQL execution exceeds the memory quota, tidb-server can spill the intermediate data of execution operators to the disk to relieve memory pressure. Operators supporting disk spill include Sort, MergeJoin, HashJoin, and HashAgg. -- The disk spill action is jointly controlled by parameters [`mem-quota-query`](/tidb-configuration-file.md#mem-quota-query), [`oom-use-tmp-storage`](/tidb-configuration-file.md#oom-use-tmp-storage), [`tmp-storage-path`](/tidb-configuration-file.md#tmp-storage-path), and [`tmp-storage-quota`](/tidb-configuration-file.md#tmp-storage-quota). +- The disk spill behavior is jointly controlled by the [`mem-quota-query`](/tidb-configuration-file.md#mem-quota-query), [`oom-use-tmp-storage`](/tidb-configuration-file.md#oom-use-tmp-storage), [`tmp-storage-path`](/tidb-configuration-file.md#tmp-storage-path), and [`tmp-storage-quota`](/tidb-configuration-file.md#tmp-storage-quota) parameters. - When the disk spill is triggered, TiDB outputs a log containing the keywords `memory exceeds quota, spill to disk now` or `memory exceeds quota, set aggregate mode to spill-mode`. -- Disk spill for operators Sort, MergeJoin, and HashJoin is new in v4.0.0; disk spill for operator HashAgg is new in v5.2.0. -- When SQL statements containing Sort, MergeJoin, or HashJoin cause OOM, TiDB triggers disk spill by default. When SQL statements containing HashAgg cause OOM, TiDB does not trigger disk spill by default. You can configure the system variable `tidb_executor_concurrency = 1` to trigger disk spill for HashAgg. +- Disk spill for the Sort, MergeJoin, and HashJoin operator is introduced in v4.0.0; disk spill for the HashAgg operator is introduced in v5.2.0. +- When the SQL executions containing Sort, MergeJoin, or HashJoin cause OOM, TiDB triggers disk spill by default. When SQL executions containing HashAgg cause OOM, TiDB does not trigger disk spill by default. You can configure the system variable `tidb_executor_concurrency = 1` to trigger disk spill for HashAgg. > **Note:** > From c7387c35894f7ccc8053cc087b7d83e0ca09feb4 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 20 Aug 2021 14:52:04 +0800 Subject: [PATCH 5/5] Update troubleshoot-write-conflicts.md --- troubleshoot-write-conflicts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot-write-conflicts.md b/troubleshoot-write-conflicts.md index 8a8feb7754f36..944c42dbba4cc 100644 --- a/troubleshoot-write-conflicts.md +++ b/troubleshoot-write-conflicts.md @@ -46,7 +46,7 @@ In the TiDB Grafana panel, check the following monitoring metrics under **KV Err ![kv-retry-duration](/media/troubleshooting-write-conflict-kv-retry-duration.png) -You can also use `[kv:9007]Write conflict` as the keywords to search in the TiDB log. The keywords also indicates the write conflict exists in the cluster. +You can also use `[kv:9007]Write conflict` as the keywords to search in the TiDB log. The keywords also indicate the write conflict exists in the cluster. ## Resolve write conflicts