Skip to content

[Spark load][Bug] Spark load's column terminator is not consistent with Broker load #4490

@xy720

Description

@xy720

Describe the bug
For Broker load:

load label test_db.label13 
( 
    DATA INFILE("hdfs://my-hdfs-host:54310/user/doris/table4") 
    into table test_tbl
    COLUMNS TERMINATED BY "|" 
    (k1,k2,name,clicks)
 ) 
WITH BROKER "doris" 
(
    "username" = "test", 
    "password" = "test"
);

For spark load:

load label test_db.label13 
( 
    DATA INFILE ("hdfs://my-hdfs-host:54310/user/doris/table4") 
    into table test 
    COLUMNS TERMINATED BY "|" 
    (k1,k2,name,clicks ) 
) 
WITH RESOURCE "spark0";

They have the same table schema、same upstream data source. The former(broker load) run successfully, but the latter(spark load) report an error "quality not good enough to cancel". And if you change the column terminator of spark load to "\|" or "\\|", then it will run successfully.

Expected behavior
We should keep the column terminator's format of spark load consistent with that of broker load.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions