[Bug] Infinite loop after using online schema change to add a new column and update values

[chh3-chan]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

0.7.0

Compute Engine

Flink 1.17.2

Minimal reproduce step

I do this all in the local environment, please let me know if i missed some key steps, many thanks!

1. Docker build percona mysql 8.0
   docker run -d --name ps -e MYSQL_ROOT_USER=root -e MYSQL_ROOT_PASSWORD=123456 -p 3306:3306 percona/percona-server:8.0
2. Add sample database inventory and table customer_percona with values
3. Install percona-toolkit 3.0.0
4. Setup and Run Flink job
   bin/flink run lib/paimon-flink-action-0.7-SNAPSHOT.jar mysql_sync_database --warehouse hdfs://localhost:8020/paimon/warehouse --database inventory --mysql_conf hostname=127.0.0.1 --mysql_conf port=3306 --mysql_conf username=root --mysql_conf password=123456 --mysql_conf database-name=inventory --mysql_conf server-time-zone=UTC --catalog_conf metastore=filesystem --table_conf bucket=4 --table_conf changelog-producer=input --table_conf sink.parallelism=4 --type_mapping to-nullable
5. Proceed online schema change to add new column
   pt-online-schema-change --alter "add column bag_fee double" h=127.0.0.1,p=123456,u=root,P=3306,D=inventory,t=customer_percona --execute
6. Update value in the new created column
   UPDATE `inventory`.`customer_percona` SET `bag_fee` = '0' WHERE (`id` = '1');

What doesn't meet your expectations?

The writer keeps showing Busy max 100%
The flink-root-taskexecutor-xxxx.log keep waiting for schema update. 2024-05-31 12:25:45,596 INFO org.apache.paimon.flink.sink.cdc.CdcRecordUtils [] - Field bag_fee not found. Waiting for schema update.

---

Before

---

Online schema change

---

After updated value in the new created column

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[MOBIN-F]

Using the pt-online-schema-change tool to perform an alter operation will generate temporary tables with suffixes of _new and _old. Paimon cannot handle this situation.

[zhaofuw]

The Flink CDC community has already improved support for the RENAME TABLE statement, and Paimon also needs to provide corresponding support.
FLINK-36610

[MOBIN-F]

The Flink CDC community has already improved support for the RENAME TABLE statement, and Paimon also needs to provide corresponding support. FLINK-36610

recommended to use flink-cdc-pipeline-paimon instead of paimon-cdc-cation