From 60905a7e71d4b266fed3bb3ab0b56195a49db441 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 6 Aug 2021 13:01:13 +0800 Subject: [PATCH 1/4] This is an automated cherry-pick of #6115 Signed-off-by: ti-chi-bot --- ticdc/manage-ticdc.md | 2 +- ticdc/troubleshoot-ticdc.md | 42 +++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/ticdc/manage-ticdc.md b/ticdc/manage-ticdc.md index 06747928d95f7..6abd999233920 100644 --- a/ticdc/manage-ticdc.md +++ b/ticdc/manage-ticdc.md @@ -858,6 +858,6 @@ In the output of the above command, if the value of `sort-engine` is "unified", > **Note:** > > + If your servers use mechanical hard drives or other storage devices that have high latency or limited bandwidth, use the unified sorter with caution. -> + The total free capacity of hard drives must be greater than or equal to 128G. If you need to replicate a large amount of historical data, make sure that the free capacity on each node is greater than or equal to the size of the incremental data that needs to be replicated. +> + The total free capacity of hard drives must be greater than or equal to 500G. If you need to replicate a large amount of historical data, make sure that the free capacity on each node is greater than or equal to the size of the incremental data that needs to be replicated. > + Unified sorter is enabled by default. If your servers do not match the above requirements and you want to disable the unified sorter, you need to manually set `sort-engine` to `memory` for the changefeed. > + To enable Unified Sorter on an existing changefeed, see the methods provided in [How do I handle the OOM that occurs after TiCDC is restarted after a task interruption?](/ticdc/troubleshoot-ticdc.md#how-do-i-handle-the-oom-that-occurs-after-ticdc-is-restarted-after-a-task-interruption). diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md index f5f4afbcb467d..f35c8fe817905 100644 --- a/ticdc/troubleshoot-ticdc.md +++ b/ticdc/troubleshoot-ticdc.md @@ -398,7 +398,49 @@ cdc cli changefeed update -c test-cf --pd=http://10.0.10.25:2379 --start-ts 4152 cdc cli changefeed resume -c test-cf --pd=http://10.0.10.25:2379 ``` +<<<<<<< HEAD > **Note:** > > + The steps above only apply to TiCDC v5.0.0 or later (excluding v5.0.0-rc). > + In v5.0.0-rc, the changefeed's checkpoint-ts after the DDL execution failure is the DDL statement's finish-ts. After using `cdc cli changefeed resume` to resume the replication task, this DDL statement will be skipped. +======= +## The default value of the time type field is inconsistent when replicating a DDL statement to the downstream MySQL 5.7. What can I do? + +Suppose that the `create table test (id int primary key, ts timestamp)` statement is executed in the upstream TiDB. When TiCDC replicates this statement to the downstream MySQL 5.7, MySQL uses the default configuration. The table schema after the replication is as follows. The default value of the `timestamp` field becomes `CURRENT_TIMESTAMP`: + +{{< copyable "sql" >}} + +```sql +mysql root@127.0.0.1:test> show create table test; ++-------+----------------------------------------------------------------------------------+ +| Table | Create Table | ++-------+----------------------------------------------------------------------------------+ +| test | CREATE TABLE `test` ( | +| | `id` int(11) NOT NULL, | +| | `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, | +| | PRIMARY KEY (`id`) | +| | ) ENGINE=InnoDB DEFAULT CHARSET=latin1 | ++-------+----------------------------------------------------------------------------------+ +1 row in set +``` + +From the result, you can see that the table schema before and after the replication is inconsistent. This is because the default value of `explicit_defaults_for_timestamp` in TiDB is different from that in MySQL. See [MySQL Compatibility](/mysql-compatibility.md#default-differences) for details. + +Since v5.0.1 or v4.0.13, for each replication to MySQL, TiCDC automatically sets `explicit_defaults_for_timestamp = ON` to ensure that the time type is consistent between the upstream and downstream. For versions earlier than v5.0.1 or v4.0.13, pay attention to the compatibility issue caused by the inconsistent `explicit_defaults_for_timestamp` value when using TiCDC to replicate the time type data. + +## When the sink of the replication downstream is TiDB or MySQL, what permissions do users of the downstream database need? + +When the sink is TiDB or MySQL, the users of the downstream database need the following permissions: + +- `Select` +- `Index` +- `Insert` +- `Update` +- `Delete` +- `Create` +- `Drop` +- `Alter` +- `Create View` + +If you need to replicate `recover table` to the downstream TiDB, the `Super` permission is required. +>>>>>>> 3c5352b03 (ticdc: add explicit_defaults_for_timestamp compatibility troubleshoot (#6115)) From bcba35d3c1b02a6bafb28faef2387d8825351912 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 6 Aug 2021 13:22:09 +0800 Subject: [PATCH 2/4] Apply suggestions from code review --- ticdc/troubleshoot-ticdc.md | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md index f35c8fe817905..b7c22e8335955 100644 --- a/ticdc/troubleshoot-ticdc.md +++ b/ticdc/troubleshoot-ticdc.md @@ -398,12 +398,12 @@ cdc cli changefeed update -c test-cf --pd=http://10.0.10.25:2379 --start-ts 4152 cdc cli changefeed resume -c test-cf --pd=http://10.0.10.25:2379 ``` -<<<<<<< HEAD > **Note:** > > + The steps above only apply to TiCDC v5.0.0 or later (excluding v5.0.0-rc). > + In v5.0.0-rc, the changefeed's checkpoint-ts after the DDL execution failure is the DDL statement's finish-ts. After using `cdc cli changefeed resume` to resume the replication task, this DDL statement will be skipped. -======= + + ## The default value of the time type field is inconsistent when replicating a DDL statement to the downstream MySQL 5.7. What can I do? Suppose that the `create table test (id int primary key, ts timestamp)` statement is executed in the upstream TiDB. When TiCDC replicates this statement to the downstream MySQL 5.7, MySQL uses the default configuration. The table schema after the replication is as follows. The default value of the `timestamp` field becomes `CURRENT_TIMESTAMP`: @@ -427,20 +427,3 @@ mysql root@127.0.0.1:test> show create table test; From the result, you can see that the table schema before and after the replication is inconsistent. This is because the default value of `explicit_defaults_for_timestamp` in TiDB is different from that in MySQL. See [MySQL Compatibility](/mysql-compatibility.md#default-differences) for details. Since v5.0.1 or v4.0.13, for each replication to MySQL, TiCDC automatically sets `explicit_defaults_for_timestamp = ON` to ensure that the time type is consistent between the upstream and downstream. For versions earlier than v5.0.1 or v4.0.13, pay attention to the compatibility issue caused by the inconsistent `explicit_defaults_for_timestamp` value when using TiCDC to replicate the time type data. - -## When the sink of the replication downstream is TiDB or MySQL, what permissions do users of the downstream database need? - -When the sink is TiDB or MySQL, the users of the downstream database need the following permissions: - -- `Select` -- `Index` -- `Insert` -- `Update` -- `Delete` -- `Create` -- `Drop` -- `Alter` -- `Create View` - -If you need to replicate `recover table` to the downstream TiDB, the `Super` permission is required. ->>>>>>> 3c5352b03 (ticdc: add explicit_defaults_for_timestamp compatibility troubleshoot (#6115)) From 88c625e27cf8306b11b5477f8b65a179af29dd50 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 6 Aug 2021 13:22:27 +0800 Subject: [PATCH 3/4] Update ticdc/troubleshoot-ticdc.md --- ticdc/troubleshoot-ticdc.md | 1 - 1 file changed, 1 deletion(-) diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md index b7c22e8335955..d38ba93f3103f 100644 --- a/ticdc/troubleshoot-ticdc.md +++ b/ticdc/troubleshoot-ticdc.md @@ -403,7 +403,6 @@ cdc cli changefeed resume -c test-cf --pd=http://10.0.10.25:2379 > + The steps above only apply to TiCDC v5.0.0 or later (excluding v5.0.0-rc). > + In v5.0.0-rc, the changefeed's checkpoint-ts after the DDL execution failure is the DDL statement's finish-ts. After using `cdc cli changefeed resume` to resume the replication task, this DDL statement will be skipped. - ## The default value of the time type field is inconsistent when replicating a DDL statement to the downstream MySQL 5.7. What can I do? Suppose that the `create table test (id int primary key, ts timestamp)` statement is executed in the upstream TiDB. When TiCDC replicates this statement to the downstream MySQL 5.7, MySQL uses the default configuration. The table schema after the replication is as follows. The default value of the `timestamp` field becomes `CURRENT_TIMESTAMP`: From 0b95392360c50074e6df217e1f62a1bdfbe6711c Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 6 Aug 2021 13:23:02 +0800 Subject: [PATCH 4/4] Update ticdc/troubleshoot-ticdc.md --- ticdc/troubleshoot-ticdc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md index d38ba93f3103f..efdcc4c330295 100644 --- a/ticdc/troubleshoot-ticdc.md +++ b/ticdc/troubleshoot-ticdc.md @@ -425,4 +425,4 @@ mysql root@127.0.0.1:test> show create table test; From the result, you can see that the table schema before and after the replication is inconsistent. This is because the default value of `explicit_defaults_for_timestamp` in TiDB is different from that in MySQL. See [MySQL Compatibility](/mysql-compatibility.md#default-differences) for details. -Since v5.0.1 or v4.0.13, for each replication to MySQL, TiCDC automatically sets `explicit_defaults_for_timestamp = ON` to ensure that the time type is consistent between the upstream and downstream. For versions earlier than v5.0.1 or v4.0.13, pay attention to the compatibility issue caused by the inconsistent `explicit_defaults_for_timestamp` value when using TiCDC to replicate the time type data. +Since v5.0.1, for each replication to MySQL, TiCDC automatically sets `explicit_defaults_for_timestamp = ON` to ensure that the time type is consistent between the upstream and downstream. For versions earlier than v5.0.1, pay attention to the compatibility issue caused by the inconsistent `explicit_defaults_for_timestamp` value when using TiCDC to replicate the time type data.