Skip to content

[Feature] Kafka_sync_database support different prefixes and suffixes for tables in different databases within one sync task #4703

@JackeyLee007

Description

@JackeyLee007

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

We have a "big" kafka topic which ingests small data from many different databases. When sinking the data into paimon ods layer, they need different prefixes and suffixes to indicate their domain. One fixed prefix or suffix could not fit this requirement.

Solution

Add two new config options: --table_prefix_db and --table_suffix_db to specify prefixes and suffixes by db.
For example:

--table_prefix_db dbname1=ods_prefix1_
--table_prefix_db dbname2=ods_prefix2_
...
--table_suffix_db dbname1=_suffix1
--table_suffix_db dbname2=_suffix2
...

The new options have higher priority than --table_prefix and --table_suffix.

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions