From b1436aa82f3dd8abdc38398ce4ba1389ff6adc47 Mon Sep 17 00:00:00 2001 From: WangXiangUSTC Date: Mon, 13 Apr 2020 00:05:57 +0800 Subject: [PATCH 1/7] Proposal: Binlog replication(syncer) unit support plugin --- docs/RFCS/20200412_syncer_plugin.md | 76 +++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 docs/RFCS/20200412_syncer_plugin.md diff --git a/docs/RFCS/20200412_syncer_plugin.md b/docs/RFCS/20200412_syncer_plugin.md new file mode 100644 index 0000000000..894c8e7f77 --- /dev/null +++ b/docs/RFCS/20200412_syncer_plugin.md @@ -0,0 +1,76 @@ +# Proposal: Binlog replication(syncer) unit support plugin + +- Author(s): [wangxiang](https://github.com/WangXiangUSTC) +- Last updated: 2020-04-12 + +## Abstract + +This proposal introduces why we need to support plugin in binlog replication(syncer) unit, and how to implements it. + +Table of contents: + +## Background + +Some users have customization requirements like below, but it is not suitable to implement in DM. We can support plugin in DM, and then user can implement their requirements by plugin. + +### Replication incompatible DDL in TiDB + +DM is a tool used to replication data from MySQL to TiDB, we know that TiDB is compatible with MySQL in most case, but some DDL is not supported in TiDB now. for example: TiDB can't reduce column's length, if you execute these SQLs in MySQL: + +```SQL +CREATE DATABASE test; +CREATE TABLE test.t1(id int primary key, name varchar(100)); +ALTER TABLE test.t1 MODIFY COLUMN name varchar(50); +``` + +And then DM will replication these SQLs to TiDB and get error `Error 1105: unsupported modify column length 50 is less than origin 100` + +DM and TiDB can't handle these SQLs now, users need to execute compatible DDL in TiDB and then skip this DDL by [binlog-event-filter](https://pingcap.com/docs/tidb-data-migration/stable/feature-overview/#binlog-event-filter) in DM. It is not convenient for users, and cannot be automated. + +In fact, the incompatible DDL `ALTER TABLE test.t1 MODIFY COLUMN name varchar(50)` can be translated to: + +```SQL +ALTER TABLE test.t1 ADD COLUMN name_tmp varchar(50) AFTER id; +REPLACE INTO test.t1(id, name_tmp) SELECT id, name AS name_tmp FROM test.t1; +ALTER TABLE test.t1 DROP COLUMN name; +ALTER TABLE test.t1 CHANGE COLUMN name_tmp name varchar(50); +``` + +Maybe we can execute them automated when meet `unsupported modify column` error. + +### Double write(just for DDL now) + +DM only support replication data to TiDB, but some users +want to send binlog to other platform while replication to TiDB. + +For example, user expect to send DDL binlog to Kafka after DDL is replication to TiDB, and then read binlog from Kafka for notifing business change. + +## Implementation + +### Interface + +For handle DDL which is not supported in TiDB or used for double write, we need to design at least two interface in plugin. + +#### Init + +We can do some initial job in the `Init` interface, for example, create connection to the downstream platform(like Kafka) for double write. + +#### HandleDDLJobResult + +This interface used to handle the DDL job's result in binlog replication unit(syncer) + +- When the ddl job execute failed, judge the error type by error code or error message, then do something to resolve it. +- When the ddl job execute success, then send it to other platform. + +### Hook + +`Init` can be execute when [initial](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L257) binlog replication unit. + +`HandleDDLJobResult` can be execute in [handleQueryEvent](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1533) after [addJobFunc](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1898) and [addJobFunc](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1705). + +## How to use + +1. User implements these interface designed in plugin +2. Build the go file in plugin mode, and generate a `.so` file +3. Set plugin in task's config file +4. Binlog replication load the plugin From c3db51147a1e243ac19c6903f53304396a97db12 Mon Sep 17 00:00:00 2001 From: WangXiangUSTC Date: Mon, 13 Apr 2020 00:08:25 +0800 Subject: [PATCH 2/7] minor update --- docs/RFCS/20200412_syncer_plugin.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/RFCS/20200412_syncer_plugin.md b/docs/RFCS/20200412_syncer_plugin.md index 894c8e7f77..b5da86763f 100644 --- a/docs/RFCS/20200412_syncer_plugin.md +++ b/docs/RFCS/20200412_syncer_plugin.md @@ -7,8 +7,6 @@ This proposal introduces why we need to support plugin in binlog replication(syncer) unit, and how to implements it. -Table of contents: - ## Background Some users have customization requirements like below, but it is not suitable to implement in DM. We can support plugin in DM, and then user can implement their requirements by plugin. From 941f2140112235057b546f1680990a7e649bece1 Mon Sep 17 00:00:00 2001 From: WangXiangUSTC Date: Mon, 13 Apr 2020 00:10:47 +0800 Subject: [PATCH 3/7] minor update --- docs/RFCS/20200412_syncer_plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RFCS/20200412_syncer_plugin.md b/docs/RFCS/20200412_syncer_plugin.md index b5da86763f..5f146971b1 100644 --- a/docs/RFCS/20200412_syncer_plugin.md +++ b/docs/RFCS/20200412_syncer_plugin.md @@ -13,7 +13,7 @@ Some users have customization requirements like below, but it is not suitable to ### Replication incompatible DDL in TiDB -DM is a tool used to replication data from MySQL to TiDB, we know that TiDB is compatible with MySQL in most case, but some DDL is not supported in TiDB now. for example: TiDB can't reduce column's length, if you execute these SQLs in MySQL: +DM is a tool used to replication data from MySQL to TiDB, we know that TiDB is compatible with MySQL in most case, but some DDL is not supported in TiDB now. For example: TiDB can't reduce column's length, if you execute these SQLs in MySQL: ```SQL CREATE DATABASE test; From 1f49e051b9bfb03238082923c5bcdf99d72888e8 Mon Sep 17 00:00:00 2001 From: WangXiangUSTC Date: Mon, 13 Apr 2020 14:19:20 +0800 Subject: [PATCH 4/7] handle dml job result --- docs/RFCS/20200412_syncer_plugin.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/RFCS/20200412_syncer_plugin.md b/docs/RFCS/20200412_syncer_plugin.md index 5f146971b1..140b63b233 100644 --- a/docs/RFCS/20200412_syncer_plugin.md +++ b/docs/RFCS/20200412_syncer_plugin.md @@ -1,7 +1,7 @@ # Proposal: Binlog replication(syncer) unit support plugin - Author(s): [wangxiang](https://github.com/WangXiangUSTC) -- Last updated: 2020-04-12 +- Last updated: 2020-04-13 ## Abstract @@ -36,7 +36,7 @@ ALTER TABLE test.t1 CHANGE COLUMN name_tmp name varchar(50); Maybe we can execute them automated when meet `unsupported modify column` error. -### Double write(just for DDL now) +### Double write DM only support replication data to TiDB, but some users want to send binlog to other platform while replication to TiDB. @@ -47,7 +47,7 @@ For example, user expect to send DDL binlog to Kafka after DDL is replication to ### Interface -For handle DDL which is not supported in TiDB or used for double write, we need to design at least two interface in plugin. +For handle DDL which is not supported in TiDB or used for double write, we need to design at least three interface in plugin. #### Init @@ -60,11 +60,20 @@ This interface used to handle the DDL job's result in binlog replication unit(sy - When the ddl job execute failed, judge the error type by error code or error message, then do something to resolve it. - When the ddl job execute success, then send it to other platform. +#### HandleDMLJobResult + +This interface used to handle the DML job's result in binlog replication unit(syncer) + +- When the dml job execute success, then send it to other platform. + ### Hook `Init` can be execute when [initial](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L257) binlog replication unit. -`HandleDDLJobResult` can be execute in [handleQueryEvent](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1533) after [addJobFunc](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1898) and [addJobFunc](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1705). +`HandleDDLJobResult` handle the result of [handleQueryEvent](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1279) and then do something. + +`HandleDMLJobResult` handle the result of [handleRowsEvent](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1274) and then do something. +`` ## How to use From 3da265a3836bb377be5f605917e27b92eb925f9a Mon Sep 17 00:00:00 2001 From: WangXiangUSTC Date: Mon, 20 Apr 2020 16:09:26 +0800 Subject: [PATCH 5/7] address comemnt --- docs/RFCS/20200412_syncer_plugin.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/RFCS/20200412_syncer_plugin.md b/docs/RFCS/20200412_syncer_plugin.md index 140b63b233..5d62580b55 100644 --- a/docs/RFCS/20200412_syncer_plugin.md +++ b/docs/RFCS/20200412_syncer_plugin.md @@ -73,7 +73,8 @@ This interface used to handle the DML job's result in binlog replication unit(sy `HandleDDLJobResult` handle the result of [handleQueryEvent](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1279) and then do something. `HandleDMLJobResult` handle the result of [handleRowsEvent](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1274) and then do something. -`` + +NOTE: We use `runFatalChan` to report errors between goroutines for executing ddl and dml, and the `err` of `handleQueryEvent` and `handleRowsEvent` don't contain the real error messag. Need to refine it. ## How to use From 9f4986d78dcc0fc7a3e28c1f171701e8007fcdf8 Mon Sep 17 00:00:00 2001 From: WangXiangUSTC Date: Mon, 11 May 2020 15:07:54 +0800 Subject: [PATCH 6/7] Update docs/RFCS/20200412_syncer_plugin.md Co-authored-by: Chunzhu Li --- docs/RFCS/20200412_syncer_plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/RFCS/20200412_syncer_plugin.md b/docs/RFCS/20200412_syncer_plugin.md index 5d62580b55..3ccef3038f 100644 --- a/docs/RFCS/20200412_syncer_plugin.md +++ b/docs/RFCS/20200412_syncer_plugin.md @@ -74,7 +74,7 @@ This interface used to handle the DML job's result in binlog replication unit(sy `HandleDMLJobResult` handle the result of [handleRowsEvent](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1274) and then do something. -NOTE: We use `runFatalChan` to report errors between goroutines for executing ddl and dml, and the `err` of `handleQueryEvent` and `handleRowsEvent` don't contain the real error messag. Need to refine it. +NOTE: We use `runFatalChan` to report errors between goroutines for executing ddl and dml, and the `err` of `handleQueryEvent` and `handleRowsEvent` don't contain the real error message. Need to refine it. ## How to use From 4e6b8641d16a15703809be9ed7c7b3f0e0fa7100 Mon Sep 17 00:00:00 2001 From: WangXiangUSTC Date: Mon, 11 May 2020 15:11:02 +0800 Subject: [PATCH 7/7] remove Note --- docs/RFCS/20200412_syncer_plugin.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/RFCS/20200412_syncer_plugin.md b/docs/RFCS/20200412_syncer_plugin.md index 3ccef3038f..d8d1eff28f 100644 --- a/docs/RFCS/20200412_syncer_plugin.md +++ b/docs/RFCS/20200412_syncer_plugin.md @@ -74,8 +74,6 @@ This interface used to handle the DML job's result in binlog replication unit(sy `HandleDMLJobResult` handle the result of [handleRowsEvent](https://github.com/pingcap/dm/blob/9023c789964fde0f5134e0c49435db557e21fdf7/syncer/syncer.go#L1274) and then do something. -NOTE: We use `runFatalChan` to report errors between goroutines for executing ddl and dml, and the `err` of `handleQueryEvent` and `handleRowsEvent` don't contain the real error message. Need to refine it. - ## How to use 1. User implements these interface designed in plugin