Skip to content

kafka load issue - Unable to load json correctly #3697

@sceneryback

Description

@sceneryback

version: apachedoris/doris-dev:build-env-1.2

Json format in kafka, not array:

{
  "id": "xxx",
  "a": {
    "id": "xxx"
  }
}

a.id is in format like '11:22:33:44:55:66'

Create Table:

CREATE TABLE `test` (
        `a_id` varchar(24) not null
        )
        DUPLICATE KEY(`a_id`)
        DISTRIBUTED BY HASH(`a_id`) BUCKETS 4
        PROPERTIES (
            "storage_type" = "COLUMN",
             "replication_num" = "1"
          );

Create load routine:

CREATE ROUTINE LOAD testdb.test_load ON test
        COLUMNS(a_id)
        PROPERTIES
        (
            "desired_concurrent_number"="3",
            "max_batch_interval" = "20",
            "max_batch_rows" = "300000",
            "max_batch_size" = "209715200",
            "strict_mode" = "false",
            "format" = "json",
            "jsonpaths" = "[\"$.a.id\"]"
        )
        FROM KAFKA
        (
            "kafka_broker_list" = "xxx",
            "kafka_topic" = "yyy"
        );

BUT results in table are:

select distinct a_id from test limit 5

+----------+
| 1 |
| 1 |
| 1 |
| 1 |
| 1 |
+----------+
Seems it only loads the first character in a.id

Metadata

Metadata

Assignees

Labels

area/loadIssues or PRs related to all kinds of loadkind/fixCategorizes issue or PR as related to a bug.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions