From 1e2af93f703d863656b2e916ece2812398304c5a Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 09:48:25 +0800 Subject: [PATCH 1/7] Create troubleshoot-ticdc.md --- ticdc/troubleshoot-ticdc.md | 67 +++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 ticdc/troubleshoot-ticdc.md diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md new file mode 100644 index 0000000000000..068763d6f6330 --- /dev/null +++ b/ticdc/troubleshoot-ticdc.md @@ -0,0 +1,67 @@ +--- +title: TiCDC FAQ and Troubleshooting +category: reference +summary: Summarizes some common issues and the solutions you might encounter when using TiCDC. +aliases: ['/docs/dev/reference/tools/ticdc/troubleshoot/'] +--- + +# TiCDC FAQ and Troubleshooting + +This document summarizes some common issues and the solutions you might encounter when using TiCDC. + +## How to select `start-ts` when starting a task? + +`start-ts` used in the replication task corresponds to a TSO in the upstream TiDB cluster. The replication task starts to request data from this TSO. Therefore, `start-ts` must meet the following requirements: + +- The value of `start-ts` is greater than the current `tikv_gc_safe_point` of the TiDB cluster; otherwise, creating a task triggers an error. +- When you start a task, make sure that all data before `start-ts` are available downstream. In some scenarios, for example, when you replicate data to message queues, you do not need to ensure the consistency of upstream and downstream data. In this case, you can relax this requirement accordingly. + +If you do not specify `start-ts` or set `start-ts=0` instead, go to PD to get a current TSO when starting a task. Then start the replication from the TSO. + +## What can I do if some tables cannot be replicated when the replication task is started? + +When you use `cdc cli changefeed create` to create a replication task, cli first checks whether the upstream tables comply with the [replication restrictions](/ticdc/ticdc-overview.md#restrictions). If there is any table that does not meet the restrictions, you can receive the message `some tables are not eligible to replicate`, with these tables listed in the message. + +If you select `Y` or `y`, the replication task continues to be created, and all updates of these tables are automatically ignored during replication. Fo other input, the replication task is not created. + +## Troubleshoot replication interrupt + +The replication interrupt might occur in the following scenarios: + +- The downstream continues to be abnormal, and TiCDC fails after many retries. + + - In this scenario, TiCDC stores the task information. Since TiCDC has set the `service GC safepoint` in PD, the data after the replication task checkpoint are not cleared by TiKV GC within the validity period of `gc-ttl`. + + - Solution: You can resume the replication task using the HTTP API after the downstream gets back to normal. + +- The replication cannot continue due to incompatible SQL statements in the downstream. + + - In this scenario, TiCDC stores the task information. Since TiCDC has set the `service GC safepoint` in PD, the data after the replication task checkpoint are not cleared by TiKV GC within the validity period of `gc-ttl`. + + - Solution: + + 1. Query the status information of the replication task by executing the `cdc cli changefeed query` command, and record the value of `checkpoint-ts`. + 2. Use the new task configuration file and add the `ignore-txn-start-ts` parameter to skip the transaction corresponding to the specified `start-ts`. + 3. Stop the old replication task using HTTP API. + 4. Execute the `cdc cli changefeed create` command. + 5. Specify the new task configuration file, and set `start-ts` to the value identical to that of `checkpoint-ts` recorded in step 1. + + Now a new replication task is started to resume replication. + +## `gc-ttl` and file sorting + +Starting from TiDB v4.0.0-rc.1, PD allows external services to set the GC safepoint. Any service can register and update its GC safepoint. PD makes sure that any KV data smaller than the GC safepoint can not be cleared by TiKV GC. TiCDC enables this feature. This ensures that the data consumed by TiCDC in TiKV cannot be cleaned up by the GC in the case of unavailable TiCDC or interrupted replication tasks. + +When starting the CDC server, you can specify the TTL of GC safepoint using `gc-ttl`. TTL represents the maximum time that TiCDC saves data smaller than the GC safepoint set by PD after the TiCDC service is stopped. The default value is `86400` (in seconds). + +If the replication task is interrupted for a long time, the accumulated unconsumed data is large. OOM might occur at the initial start of TiCDC. In this case, you can enable the file sorting feature provided by TiCDC. This feature sorts files using the file system. To enable this feature, you need to pass `--sort-engine=file` and `--sort-dir=/path/to/sort_dir` into `cdc cli` when creating a replication task. An example is as follows: + +{{< copyable "shell-regular" >}} + +```shell +cdc cli changefeed create --pd=http://10.0.10.25:2379 --start-ts=415238226621235200 --sink-uri="mysql://root:123456@127.0.0.1:3306/" --sort-engine="file" --sort-dir="/data/cdc/sort" +``` + +> **Note:** +> +> TiCDC v4.0 does not support dynamic modification of file sorting and memory sorting. From ba8dc8e4ebdb927a2d761ded822de5dbe0e8faf9 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 09:52:27 +0800 Subject: [PATCH 2/7] remove docs --- ticdc/column-ddl-type-codes.md | 84 ---------------------------------- ticdc/sink-url.md | 59 ------------------------ ticdc/troubleshoot-ticdc.md | 6 +-- 3 files changed, 3 insertions(+), 146 deletions(-) delete mode 100644 ticdc/column-ddl-type-codes.md delete mode 100644 ticdc/sink-url.md diff --git a/ticdc/column-ddl-type-codes.md b/ticdc/column-ddl-type-codes.md deleted file mode 100644 index 602552f79e3cb..0000000000000 --- a/ticdc/column-ddl-type-codes.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: Column and DDL Type Codes -summary: Learn the column and DDL type codes. -category: reference -aliases: ['/docs/dev/reference/tools/ticdc/column-ddl-type/'] ---- - -# Column and DDL Type Codes - -The column and DDL type codes are encodings defined by [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md). The column type code identifies the column data type in Row Changed Events. The DDL type code identifies the DDL statement type in DDL Events. - -## Column Type Code - -| Type | Code | Output example | Note | -| :---------- | :--- | :------ | :-- | -| Decimal | 0 | {"t":0,"v":"129012.1230000"} | | -| Tiny/Bool | 1 | {"t":1,"v":1} | | -| Short | 2 | {"t":2,"v":1} | | -| Long | 3 | {"t":3,"v":123} | | -| Float | 4 | {"t":4,"v":153.123} | | -| Double | 5 | {"t":5,"v":153.123} | | -| Null | 6 | {"t":6,"v":null} | | -| Timestamp | 7 | {"t":7,"v":"1973-12-30 15:30:00"} | | -| Longlong | 8 | {"t":8,"v":123} | | -| Int24 | 9 | {"t":9,"v":123} | | -| Date | 10 | {"t":10,"v":"2000-01-01"} | | -| Duration | 11 | {"t":11,"v":"23:59:59"} | | -| Datetime | 12 | {"t":12,"v":"2015-12-20 23:58:58"} | | -| Year | 13 | {"t":13,"v":1970} | | -| New Date | 14 | {"t":14,"v":"2000-01-01"} | | -| Varchar | 15 | {"t":15,"v":"test"} | The value is encoded in UTF-8. | -| Bit | 16 | {"t":16,"v":81} | | -| JSON | 245 | {"t":245,"v":"{\\"key1\\": \\"value1\\"}"} | | -| New Decimal | 246 | {"t":246,"v":"129012.1230000"} | | -| Enum | 247 | {"t":247,"v":1} | | -| Set | 248 | {"t":248,"v":3} | | -| Tiny Blob | 249 | {"t":249,"v":"5rWL6K+VdGV4dA=="} | The value is encoded in Base64. | -| Medium Blob | 250 | {"t":250,"v":"5rWL6K+VdGV4dA=="} | The value is encoded in Base64. | -| Long Blob | 251 | {"t":251,"v":"5rWL6K+VdGV4dA=="} | The value is encoded in Base64. | -| Blob | 252 | {"t":252,"v":"5rWL6K+VdGV4dA=="} | The value is encoded in Base64. | -| Var String | 253 | {"t":253,"v":"test"} | The value is encoded in UTF-8. | -| String | 254 | {"t":254,"v":"test"} | The value is encoded in UTF-8. | -| Geometry | 255 | | Unsupported type | - -## DDL Type Code - -| Type | Code | -| :-------------------------------- | :- | -| Create Schema | 1 | -| Drop Schema | 2 | -| Create Table | 3 | -| Drop Table | 4 | -| Add Column | 5 | -| Drop Column | 6 | -| Add Index | 7 | -| Drop Index | 8 | -| Add Foreign Key | 9 | -| Drop Foreign Key | 10 | -| Truncate Table | 11 | -| Modify Column | 12 | -| Rebase Auto ID | 13 | -| Rename Table | 14 | -| Set Default Value | 15 | -| Shard RowID | 16 | -| Modify Table Comment | 17 | -| Rename Index | 18 | -| Add Table Partition | 19 | -| Drop Table Partition | 20 | -| Create View | 21 | -| Modify Table Charset And Collate | 22 | -| Truncate Table Partition | 23 | -| Drop View | 24 | -| Recover Table | 25 | -| Modify Schema Charset And Collate | 26 | -| Lock Table | 27 | -| Unlock Table | 28 | -| Repair Table | 29 | -| Set TiFlash Replica | 30 | -| Update TiFlash Replica Status | 31 | -| Add Primary Key | 32 | -| Drop Primary Key | 33 | -| Create Sequence | 34 | -| Alter Sequence | 35 | -| Drop Sequence | 36 | diff --git a/ticdc/sink-url.md b/ticdc/sink-url.md deleted file mode 100644 index 19168ff08c7c4..0000000000000 --- a/ticdc/sink-url.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -title: Configure Sink URI -summary: Learn how to configure sink URI. -category: reference -aliases: ['/docs/dev/reference/tools/ticdc/sink/'] ---- - -# Configure Sink URI - -You need to configure sink URI in the following format. Currently, the `scheme` supports `mysql`, `tidb`, and `kafka`. - -{{< copyable "" >}} - -``` -[scheme]://[userinfo@][host]:[port][/path]?[query_parameters] -``` - -## Configure sink URI with `mysql` or `tidb` - -The following command is a configuration example: - -{{< copyable "shell-regular" >}} - -```shell ---sink-uri="mysql://root:123456@127.0.0.1:3306/?worker-count=16&max-txn-row=5000" -``` - -The parameters in the above command are described as follows: - -| Parameter | Description | -| :------------ | :------------------------------------------------ | -| `root` | The username of the downstream database | -| `123456` | The password of the downstream database | -| `127.0.0.1` | The IP of the downstream database | -| `3306` | The port of the downstream database | -| `worker-count` | The number of SQL statements that can be concurrently executed to the downstream (optional, `16` by default) | -| `max-txn-row` | The size of a transaction batch that can be executed to the downstream (optional, `256` by default) | - -## Configure sink URI with `kafka` - -The following command is an configuration example: - -{{< copyable "shell-regular" >}} - -```shell ---sink-uri="kafka://127.0.0.1:9092/cdc-test?kafka-version=2.4.0&partition-num=6&max-message-bytes=67108864&replication-factor=1" -``` - -The parameters in the above command are described as follows: - -| Parameter | Description | -| :------------------ | :------------------------------------------------------------ | -| `127.0.0.1` | The IP of downstream Kafka services | -| `9092` | The port of downstream Kafka | -| `cdc-test` | The name of the Kafka topic | -| `kafka-version` | The version of downstream Kafka (optional, `2.4.0` by default) | -| `partition-num` | The number of downstream Kafka partitions (optional. The value must be **no greater than** the actual number of partitions. If you do not configure this parameter, the partition number will be obtained automatically.) | -| `max-message-bytes` | The maximum size of data that is sent to Kafka broker each time (optional, `64MB` by default) | -| `replication-factor` | The number of Kafka message replicas that can be saved (optional, `1` by default) | diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md index 068763d6f6330..0b804a9136b7e 100644 --- a/ticdc/troubleshoot-ticdc.md +++ b/ticdc/troubleshoot-ticdc.md @@ -9,7 +9,7 @@ aliases: ['/docs/dev/reference/tools/ticdc/troubleshoot/'] This document summarizes some common issues and the solutions you might encounter when using TiCDC. -## How to select `start-ts` when starting a task? +## How to select `start-ts` when starting a task `start-ts` used in the replication task corresponds to a TSO in the upstream TiDB cluster. The replication task starts to request data from this TSO. Therefore, `start-ts` must meet the following requirements: @@ -18,7 +18,7 @@ This document summarizes some common issues and the solutions you might encounte If you do not specify `start-ts` or set `start-ts=0` instead, go to PD to get a current TSO when starting a task. Then start the replication from the TSO. -## What can I do if some tables cannot be replicated when the replication task is started? +## Some tables cannot be replicated when the replication task is started When you use `cdc cli changefeed create` to create a replication task, cli first checks whether the upstream tables comply with the [replication restrictions](/ticdc/ticdc-overview.md#restrictions). If there is any table that does not meet the restrictions, you can receive the message `some tables are not eligible to replicate`, with these tables listed in the message. @@ -52,7 +52,7 @@ The replication interrupt might occur in the following scenarios: Starting from TiDB v4.0.0-rc.1, PD allows external services to set the GC safepoint. Any service can register and update its GC safepoint. PD makes sure that any KV data smaller than the GC safepoint can not be cleared by TiKV GC. TiCDC enables this feature. This ensures that the data consumed by TiCDC in TiKV cannot be cleaned up by the GC in the case of unavailable TiCDC or interrupted replication tasks. -When starting the CDC server, you can specify the TTL of GC safepoint using `gc-ttl`. TTL represents the maximum time that TiCDC saves data smaller than the GC safepoint set by PD after the TiCDC service is stopped. The default value is `86400` (in seconds). +When starting the CDC server, you can specify the TTL of GC safepoint using `gc-ttl`. TTL represents the maximum time that TiCDC saves the data smaller than the GC safepoint set by PD after the TiCDC service is stopped. The default value is `86400` (in seconds). If the replication task is interrupted for a long time, the accumulated unconsumed data is large. OOM might occur at the initial start of TiCDC. In this case, you can enable the file sorting feature provided by TiCDC. This feature sorts files using the file system. To enable this feature, you need to pass `--sort-engine=file` and `--sort-dir=/path/to/sort_dir` into `cdc cli` when creating a replication task. An example is as follows: From f2dbf413b4fc79559e835a98342e59e1b53a9255 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 09:57:41 +0800 Subject: [PATCH 3/7] add faq to ticdc-overview --- ticdc/ticdc-overview.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/ticdc/ticdc-overview.md b/ticdc/ticdc-overview.md index a65bd1d0fe54d..49852179263da 100644 --- a/ticdc/ticdc-overview.md +++ b/ticdc/ticdc-overview.md @@ -67,6 +67,10 @@ Currently, The following scenarios are not supported: For details, see [Manage TiCDC Cluster and Replication Tasks](/ticdc/manage-ticdc.md). +## TiCDC FAQ and Troubleshooting + +For details, refer to [FAQ and Troubleshooting](/ticdc/troubleshoot-ticdc.md). + ## TiCDC Open Protocol TiCDC Open Protocol is a row-level data change notification protocol that provides data sources for monitoring, caching, full-text indexing, analysis engines, and master-slave replication between different databases. TiCDC complies with TiCDC Open Protocol and replicates data changes of TiDB to third-party data medium such as MQ (Message Queue). For more information, see [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md). From 98e17f9ae7c90999001901e4a422601239d20f82 Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Fri, 12 Jun 2020 10:09:15 +0800 Subject: [PATCH 4/7] Update TOC.md --- TOC.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/TOC.md b/TOC.md index e386ef62acb81..8a373b5bba814 100644 --- a/TOC.md +++ b/TOC.md @@ -64,7 +64,7 @@ + [BR Use Cases](/br/backup-and-restore-use-cases.md) + [BR storages](/br/backup-and-restore-storages.md) + [Daily Checklist](/daily-check.md) - + [Maintain TiCDC](/ticdc/manage-ticdc.md) + + [Manage TiCDC Cluster and Replication Tasks](/ticdc/manage-ticdc.md) + [Maintain TiFlash](/tiflash/maintain-tiflash.md) + [Maintain TiDB Using TiUP](/maintain-tidb-using-tiup.md) + [Maintain TiDB Using Ansible](/maintain-tidb-using-ansible.md) @@ -79,6 +79,7 @@ + [Statement Summary Tables](/statement-summary-tables.md) + [Troubleshoot Cluster Setup](/troubleshoot-tidb-cluster.md) + [TiDB Troubleshooting Map](/tidb-troubleshooting-map.md) + + [TiCDC FAQ and Troubleshooting](/ticdc/troubleshoot-ticdc.md) + [Troubleshoot TiFlash](/tiflash/troubleshoot-tiflash.md) + Performance Tuning + Software Tuning @@ -149,11 +150,7 @@ + [Troubleshoot](/troubleshoot-tidb-lightning.md) + [FAQ](/tidb-lightning/tidb-lightning-faq.md) + [Glossary](/tidb-lightning/tidb-lightning-glossary.md) - + TiCDC - + [Overview](/ticdc/ticdc-overview.md) - + [Configure Sink URI](/ticdc/sink-url.md) - + [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md) - + [Column and DDL Type Codes](/ticdc/column-ddl-type-codes.md) + + [TiCDC](/ticdc/ticdc-overview.md) + sync-diff-inspector + [Overview](/sync-diff-inspector/sync-diff-inspector-overview.md) + [Data Check for Tables with Different Schema/Table Names](/sync-diff-inspector/route-diff.md) @@ -389,6 +386,8 @@ + [tiup-mirror](/tiup/tiup-mirrors.md) + [tiup-bench](/tiup/tiup-bench.md) + [Errors Codes](/error-codes.md) + + [TiCDC Overview](/ticdc/ticdc-overview.md) + + [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md) + FAQs + [TiDB FAQs](/faq/tidb-faq.md) + [TiDB Lightning FAQs](/tidb-lightning/tidb-lightning-faq.md) From 772fef839b260f40c5a5211dbd8563733c937fac Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Mon, 15 Jun 2020 21:48:15 +0800 Subject: [PATCH 5/7] Update ticdc-overview.md --- ticdc/ticdc-overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ticdc/ticdc-overview.md b/ticdc/ticdc-overview.md index 208259bdf52a9..dfe58e4f32751 100644 --- a/ticdc/ticdc-overview.md +++ b/ticdc/ticdc-overview.md @@ -67,9 +67,9 @@ Currently, The following scenarios are not supported: For details, see [Manage TiCDC Cluster and Replication Tasks](/ticdc/manage-ticdc.md). -## TiCDC FAQ and Troubleshooting +## Troubleshoot TiCDC -For details, refer to [FAQ and Troubleshooting](/ticdc/troubleshoot-ticdc.md). +For details, refer to [Troubleshoot TiCDC](/ticdc/troubleshoot-ticdc.md). ## TiCDC Open Protocol From 92a1cccf986986acec0bbfed6d3ad46bf14335dc Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Mon, 15 Jun 2020 21:49:19 +0800 Subject: [PATCH 6/7] Update troubleshoot-ticdc.md --- ticdc/troubleshoot-ticdc.md | 1 - 1 file changed, 1 deletion(-) diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md index 2de9b36080d44..995075c931038 100644 --- a/ticdc/troubleshoot-ticdc.md +++ b/ticdc/troubleshoot-ticdc.md @@ -55,4 +55,3 @@ cdc cli changefeed create --pd=http://10.0.10.25:2379 --start-ts=415238226621235 > **Note:** > > TiCDC (the 4.0 version) does not support dynamically modifying the file sorting and memory sorting yet. - From 14c6f8ade3f4e196140878cca60c16cc9cd3b66d Mon Sep 17 00:00:00 2001 From: toutdesuite Date: Mon, 15 Jun 2020 22:04:52 +0800 Subject: [PATCH 7/7] Update daily-check.md --- daily-check.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daily-check.md b/daily-check.md index 314cb7310a99e..c118ebd015515 100644 --- a/daily-check.md +++ b/daily-check.md @@ -26,7 +26,7 @@ TiDB Dashboard simplifies the operation and maintenance of the TiDB database. Yo ![Host panel](/media/host-panel.png) -You can view the usage of CPU, memory, and disk. When the average usage of any resource exceeds 60%, it is recommended to plan to scale out the capacity. When the average usage reaches 80%, it is recommended to scale out the capacity. +You can view the usage of CPU, memory, and disk. When the usage of any resource exceeds 80%, it is recommended to scale out the capacity accordingly. ### SQL analysis panel