Skip to content

Missing dimensions/metrics are parsed as empty values for delimited text files #8778

@a2l007

Description

@a2l007

Affected Version

0.14.0-incubating. Also reproducible in master

Description

With -Ddruid.generic.useDefaultValueForNull=false nulls and empty values should be handled differently. However for delimited text files, it looks like null/missing values are being parsed as empty values.
For example with the following dimension & metric spec and CSV data:

 "dimensionsSpec" : {
            "dimensions" : [ "dim1", "dim2", "dim3"]
          }

"metricsSpec" : [
        {
          "name" : "metric1",
          "type" : "count",
           "fieldName" : "metric1"
       },
        {
          "name" : "metric2",
          "type" : "longSum",
          "fieldName" : "metric2"
        }
      ]

Data:

2019-07-04T03:46:58.771Z,,"value2","value3",,82

dim1 would be stored as "" instead of null and metric1 would be stored as 0 instead of null even though they are missing values.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions