doris-flink-connecter数据换行配置不起作用

[yangmingjie]

Search before asking

• I had searched in the issues and found no similar issues.

Version

1.0.0

What's Wrong?

在使用doris-connector的时候，数据导入到doris一部分的试试报错：

最后发现问题如下：

mysql数据库中有个字段mateName是varchar（255），如下：

拷贝出来后内容，有换行：

然后，

但是 还是报一次样的错误。

What You Expected?

希望设置换行符 sink.properties.line_delimiter 和 sink.properties.column_separator 能起作用。或者其他解决这个问题的办法。

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[JNSimba]

#6497

[minchowang]

@JNSimba Has this problem been solved?

[JNSimba]

@JNSimba Has this problem been solved?

Specify the newline character to set the
sink.properties.line_delimiter parameters

[minchowang]

@JNSimba thanks for your reply. here is my code, and some errors.
doris ddl：

CREATE TABLE `product` (
  `id` int(11) NULL COMMENT "",
...
  `gmt_modified` datetime NULL COMMENT "",
  `gmt_create` datetime NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(`id`, `id_hash`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"storage_format" = "V2"
);

flink sql doris-sink ddl：

CREATE TABLE product (
 id INT,
...
 gmt_modified TIMESTAMP(6),
 gmt_create TIMESTAMP(6) ) 
WITH (  
 'connector' = 'doris', 
 'fenodes' = 'n3:8031', 
  'table.identifier' = 'dev.product',  
  'username' = 'root',
  'sink.batch.size' = '100',
  'sink.properties.column_separator' = '\x01',
  'sink.properties.line_delimiter' = '\x02',
  'sink.batch.interval' = '2s' , 
  'password' = 'doris'
);

# flink insert sql
insert into product 
select * from product_cdc_table ;

When we write data to dorsi by flink streaming, have following error occurs.

[minchowang]

@JNSimba thanks for your reply. here is my code, and some errors.
doris ddl：

CREATE TABLE `product` (
  `id` int(11) NULL COMMENT "",
...
  `gmt_modified` datetime NULL COMMENT "",
  `gmt_create` datetime NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(`id`, `id_hash`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"storage_format" = "V2"
);

flink sql doris-sink ddl：

CREATE TABLE product (
 id INT,
...
 gmt_modified TIMESTAMP(6),
 gmt_create TIMESTAMP(6) ) 
WITH (  
 'connector' = 'doris', 
 'fenodes' = 'n3:8031', 
  'table.identifier' = 'dev.product',  
  'username' = 'root',
  'sink.batch.size' = '100',
  'sink.properties.column_separator' = '\x01',
  'sink.properties.line_delimiter' = '\x02',
  'sink.batch.interval' = '2s' , 
  'password' = 'doris'
);

# flink insert sql
insert into product 
select * from product_cdc_table ;

When we write data to dorsi by flink streaming, have following error occurs.

doris version: 0.14
flink-doris-connector version : master
flink version ： 1.12.2

When we repalce column_separator=%x01 and line_delimiter=%x02, have other error occurs :

Reason: actual column number is more than schema column number. actual number: 1900 sep: %, schema number: 19; . src line: [689%x018a7c7089faaf00c850b90154e74fb821%x01101787%x0111%x010%x010%x010%x01%x01%x01QARFEE 

We check barch-0.14 and master, look the commit message [Load] Support multi bytes LineDelimiter and ColumnSeparator (#5462) in broker_scanner.cpp, so we are going to verify it.

[yangmingjie]

@JNSimba Has this problem been solved?

Specify the newline character to set the
sink.properties.line_delimiter parameters

我这里试了还是不行，最近一忙就没顾上 @mincwang

[JNSimba]

@JNSimba Has this problem been solved?

Specify the newline character to set the
sink.properties.line_delimiter parameters

我这里试了还是不行，最近一忙就没顾上 @mincwang

@yangmingjie your param line_delimiter(%%%%) and column_separator(%%%%) has the same prefix value, may cause doris to be indistinguishable，modify them to a different prefix and it works

[minchowang]

@JNSimba thanks for your reply. here is my code, and some errors.
doris ddl：

CREATE TABLE `product` (
  `id` int(11) NULL COMMENT "",
...
  `gmt_modified` datetime NULL COMMENT "",
  `gmt_create` datetime NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(`id`, `id_hash`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"storage_format" = "V2"
);

flink sql doris-sink ddl：

CREATE TABLE product (
 id INT,
...
 gmt_modified TIMESTAMP(6),
 gmt_create TIMESTAMP(6) ) 
WITH (  
 'connector' = 'doris', 
 'fenodes' = 'n3:8031', 
  'table.identifier' = 'dev.product',  
  'username' = 'root',
  'sink.batch.size' = '100',
  'sink.properties.column_separator' = '\x01',
  'sink.properties.line_delimiter' = '\x02',
  'sink.batch.interval' = '2s' , 
  'password' = 'doris'
);

# flink insert sql
insert into product 
select * from product_cdc_table ;

When we write data to dorsi by flink streaming, have following error occurs.

doris version: 0.14
flink-doris-connector version : master
flink version ： 1.12.2

When we repalce column_separator=%x01 and line_delimiter=%x02, have other error occurs :

Reason: actual column number is more than schema column number. actual number: 1900 sep: %, schema number: 19; . src line: [689%x018a7c7089faaf00c850b90154e74fb821%x01101787%x0111%x010%x010%x010%x01%x01%x01QARFEE 

We check barch-0.14 and master, look the commit message [Load] Support multi bytes LineDelimiter and ColumnSeparator (#5462) in broker_scanner.cpp, so we are going to verify it.

@JNSimba Can this PR fix the above problems? please help to confirm.

[JNSimba]

@mincwang it is work to change to other characters，hexadecimal may be has problem

[minchowang]

@mincwang it is work to change to other characters，hexadecimal may be has problem

What do you recommend set for column_separator and line_delimiter ?

[JNSimba]

@mincwang it is work to change to other characters，hexadecimal may be has problem

What do you recommend set for column_separator and line_delimiter ?

It can be set as a character that does not appear frequently according to the specific business, or it can be set as a combination of multiple characters

[minchowang]

@mincwang it is work to change to other characters，hexadecimal may be has problem

What do you recommend set for column_separator and line_delimiter ?

It can be set as a character that does not appear frequently according to the specific business, or it can be set as a combination of multiple characters

But we doris version is 0.14，only master version support of multiple characters ，fix above problem by this PR #5462.

[JNSimba]

@mincwang it is work to change to other characters，hexadecimal may be has problem

What do you recommend set for column_separator and line_delimiter ?

It can be set as a character that does not appear frequently according to the specific business, or it can be set as a combination of multiple characters

But we doris version is 0.14，only master version support of multiple characters ，fix above problem by this PR #5462.

Sorry, I may not make it clear, I mean it can be specified as multiple letters, like column_separator=doris