From 685ddb2b1f89b8ca0d49c038da73b081149d2285 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 6 Aug 2021 13:01:13 +0800 Subject: [PATCH 1/3] This is an automated cherry-pick of #6115 Signed-off-by: ti-chi-bot --- ticdc/manage-ticdc.md | 5 +++++ ticdc/troubleshoot-ticdc.md | 42 +++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/ticdc/manage-ticdc.md b/ticdc/manage-ticdc.md index 5cde7ae6671ea..5f6a906a05f3b 100644 --- a/ticdc/manage-ticdc.md +++ b/ticdc/manage-ticdc.md @@ -810,6 +810,11 @@ In the output of the above command, if the value of `sort-engine` is "unified", > **Note:** > > + If your servers use mechanical hard drives or other storage devices that have high latency or limited bandwidth, use the unified sorter with caution. +<<<<<<< HEAD > + The total free capacity of hard drives must be greater than or equal to 128G. If you need to replicate a large amount of historical data, make sure that the free capacity on each node is greater than or equal to the size of the incremental data that needs to be replicated. > + Since v4.0.13, Unified sorter is enabled by default. If your servers do not match the above requirements and you want to disable the unified sorter, you need to manually set `sort-engine` to `memory` for the changefeed. +======= +> + The total free capacity of hard drives must be greater than or equal to 500G. If you need to replicate a large amount of historical data, make sure that the free capacity on each node is greater than or equal to the size of the incremental data that needs to be replicated. +> + Unified sorter is enabled by default. If your servers do not match the above requirements and you want to disable the unified sorter, you need to manually set `sort-engine` to `memory` for the changefeed. +>>>>>>> 3c5352b03 (ticdc: add explicit_defaults_for_timestamp compatibility troubleshoot (#6115)) > + To enable Unified Sorter on an existing changefeed, see the methods provided in [How do I handle the OOM that occurs after TiCDC is restarted after a task interruption?](/ticdc/troubleshoot-ticdc.md#how-do-i-handle-the-oom-that-occurs-after-ticdc-is-restarted-after-a-task-interruption). diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md index e34c34ebb49b9..f29ba9e88c024 100644 --- a/ticdc/troubleshoot-ticdc.md +++ b/ticdc/troubleshoot-ticdc.md @@ -400,7 +400,49 @@ cdc cli changefeed update -c test-cf --pd=http://10.0.10.25:2379 --start-ts 4152 cdc cli changefeed resume -c test-cf --pd=http://10.0.10.25:2379 ``` +<<<<<<< HEAD > **Note:** > > + The steps above only apply to TiCDC v4.0.11 or later. > + In versions earlier than v4.0.11, the changefeed's checkpoint-ts after the DDL execution failure is the DDL statement's finish-ts. After using `cdc cli changefeed resume` to resume the replication task, this DDL statement will be skipped. +======= +## The default value of the time type field is inconsistent when replicating a DDL statement to the downstream MySQL 5.7. What can I do? + +Suppose that the `create table test (id int primary key, ts timestamp)` statement is executed in the upstream TiDB. When TiCDC replicates this statement to the downstream MySQL 5.7, MySQL uses the default configuration. The table schema after the replication is as follows. The default value of the `timestamp` field becomes `CURRENT_TIMESTAMP`: + +{{< copyable "sql" >}} + +```sql +mysql root@127.0.0.1:test> show create table test; ++-------+----------------------------------------------------------------------------------+ +| Table | Create Table | ++-------+----------------------------------------------------------------------------------+ +| test | CREATE TABLE `test` ( | +| | `id` int(11) NOT NULL, | +| | `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, | +| | PRIMARY KEY (`id`) | +| | ) ENGINE=InnoDB DEFAULT CHARSET=latin1 | ++-------+----------------------------------------------------------------------------------+ +1 row in set +``` + +From the result, you can see that the table schema before and after the replication is inconsistent. This is because the default value of `explicit_defaults_for_timestamp` in TiDB is different from that in MySQL. See [MySQL Compatibility](/mysql-compatibility.md#default-differences) for details. + +Since v5.0.1 or v4.0.13, for each replication to MySQL, TiCDC automatically sets `explicit_defaults_for_timestamp = ON` to ensure that the time type is consistent between the upstream and downstream. For versions earlier than v5.0.1 or v4.0.13, pay attention to the compatibility issue caused by the inconsistent `explicit_defaults_for_timestamp` value when using TiCDC to replicate the time type data. + +## When the sink of the replication downstream is TiDB or MySQL, what permissions do users of the downstream database need? + +When the sink is TiDB or MySQL, the users of the downstream database need the following permissions: + +- `Select` +- `Index` +- `Insert` +- `Update` +- `Delete` +- `Create` +- `Drop` +- `Alter` +- `Create View` + +If you need to replicate `recover table` to the downstream TiDB, the `Super` permission is required. +>>>>>>> 3c5352b03 (ticdc: add explicit_defaults_for_timestamp compatibility troubleshoot (#6115)) From 6dddf0cc7b05468b05d87a0314cd72da96f81afd Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 6 Aug 2021 13:20:05 +0800 Subject: [PATCH 2/3] Apply suggestions from code review --- ticdc/manage-ticdc.md | 7 +------ ticdc/troubleshoot-ticdc.md | 21 +-------------------- 2 files changed, 2 insertions(+), 26 deletions(-) diff --git a/ticdc/manage-ticdc.md b/ticdc/manage-ticdc.md index 5f6a906a05f3b..fd635ba571484 100644 --- a/ticdc/manage-ticdc.md +++ b/ticdc/manage-ticdc.md @@ -810,11 +810,6 @@ In the output of the above command, if the value of `sort-engine` is "unified", > **Note:** > > + If your servers use mechanical hard drives or other storage devices that have high latency or limited bandwidth, use the unified sorter with caution. -<<<<<<< HEAD -> + The total free capacity of hard drives must be greater than or equal to 128G. If you need to replicate a large amount of historical data, make sure that the free capacity on each node is greater than or equal to the size of the incremental data that needs to be replicated. -> + Since v4.0.13, Unified sorter is enabled by default. If your servers do not match the above requirements and you want to disable the unified sorter, you need to manually set `sort-engine` to `memory` for the changefeed. -======= > + The total free capacity of hard drives must be greater than or equal to 500G. If you need to replicate a large amount of historical data, make sure that the free capacity on each node is greater than or equal to the size of the incremental data that needs to be replicated. -> + Unified sorter is enabled by default. If your servers do not match the above requirements and you want to disable the unified sorter, you need to manually set `sort-engine` to `memory` for the changefeed. ->>>>>>> 3c5352b03 (ticdc: add explicit_defaults_for_timestamp compatibility troubleshoot (#6115)) +> + Since v4.0.13, Unified sorter is enabled by default. If your servers do not match the above requirements and you want to disable the unified sorter, you need to manually set `sort-engine` to `memory` for the changefeed. > + To enable Unified Sorter on an existing changefeed, see the methods provided in [How do I handle the OOM that occurs after TiCDC is restarted after a task interruption?](/ticdc/troubleshoot-ticdc.md#how-do-i-handle-the-oom-that-occurs-after-ticdc-is-restarted-after-a-task-interruption). diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md index f29ba9e88c024..db6ac91c2e94b 100644 --- a/ticdc/troubleshoot-ticdc.md +++ b/ticdc/troubleshoot-ticdc.md @@ -400,12 +400,10 @@ cdc cli changefeed update -c test-cf --pd=http://10.0.10.25:2379 --start-ts 4152 cdc cli changefeed resume -c test-cf --pd=http://10.0.10.25:2379 ``` -<<<<<<< HEAD > **Note:** > > + The steps above only apply to TiCDC v4.0.11 or later. > + In versions earlier than v4.0.11, the changefeed's checkpoint-ts after the DDL execution failure is the DDL statement's finish-ts. After using `cdc cli changefeed resume` to resume the replication task, this DDL statement will be skipped. -======= ## The default value of the time type field is inconsistent when replicating a DDL statement to the downstream MySQL 5.7. What can I do? Suppose that the `create table test (id int primary key, ts timestamp)` statement is executed in the upstream TiDB. When TiCDC replicates this statement to the downstream MySQL 5.7, MySQL uses the default configuration. The table schema after the replication is as follows. The default value of the `timestamp` field becomes `CURRENT_TIMESTAMP`: @@ -428,21 +426,4 @@ mysql root@127.0.0.1:test> show create table test; From the result, you can see that the table schema before and after the replication is inconsistent. This is because the default value of `explicit_defaults_for_timestamp` in TiDB is different from that in MySQL. See [MySQL Compatibility](/mysql-compatibility.md#default-differences) for details. -Since v5.0.1 or v4.0.13, for each replication to MySQL, TiCDC automatically sets `explicit_defaults_for_timestamp = ON` to ensure that the time type is consistent between the upstream and downstream. For versions earlier than v5.0.1 or v4.0.13, pay attention to the compatibility issue caused by the inconsistent `explicit_defaults_for_timestamp` value when using TiCDC to replicate the time type data. - -## When the sink of the replication downstream is TiDB or MySQL, what permissions do users of the downstream database need? - -When the sink is TiDB or MySQL, the users of the downstream database need the following permissions: - -- `Select` -- `Index` -- `Insert` -- `Update` -- `Delete` -- `Create` -- `Drop` -- `Alter` -- `Create View` - -If you need to replicate `recover table` to the downstream TiDB, the `Super` permission is required. ->>>>>>> 3c5352b03 (ticdc: add explicit_defaults_for_timestamp compatibility troubleshoot (#6115)) +Since v4.0.13, for each replication to MySQL, TiCDC automatically sets `explicit_defaults_for_timestamp = ON` to ensure that the time type is consistent between the upstream and downstream. For versions earlier than v4.0.13, pay attention to the compatibility issue caused by the inconsistent `explicit_defaults_for_timestamp` value when using TiCDC to replicate the time type data. From e7f02b35c616ac81f92e6a733a7192fb12eafca4 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 6 Aug 2021 13:20:34 +0800 Subject: [PATCH 3/3] Update ticdc/troubleshoot-ticdc.md --- ticdc/troubleshoot-ticdc.md | 1 + 1 file changed, 1 insertion(+) diff --git a/ticdc/troubleshoot-ticdc.md b/ticdc/troubleshoot-ticdc.md index db6ac91c2e94b..90b05a93bcdbe 100644 --- a/ticdc/troubleshoot-ticdc.md +++ b/ticdc/troubleshoot-ticdc.md @@ -404,6 +404,7 @@ cdc cli changefeed resume -c test-cf --pd=http://10.0.10.25:2379 > > + The steps above only apply to TiCDC v4.0.11 or later. > + In versions earlier than v4.0.11, the changefeed's checkpoint-ts after the DDL execution failure is the DDL statement's finish-ts. After using `cdc cli changefeed resume` to resume the replication task, this DDL statement will be skipped. + ## The default value of the time type field is inconsistent when replicating a DDL statement to the downstream MySQL 5.7. What can I do? Suppose that the `create table test (id int primary key, ts timestamp)` statement is executed in the upstream TiDB. When TiCDC replicates this statement to the downstream MySQL 5.7, MySQL uses the default configuration. The table schema after the replication is as follows. The default value of the `timestamp` field becomes `CURRENT_TIMESTAMP`: