Cell contains [LJava.lang.String@ instead of data when constructing array via transformSpec during ingestion

[fstolba]

Affected Version

0.16.0

Description

The cluster runs as single-server-large. I'm trying to combine multiple fields into an array during native batch ingestion.

The task spec looks like this:

{
  "type": "index_parallel",
  "spec": {
    "ioConfig" : {
      "type": "index_parallel",
      "firehose" : {
        "type"    : "ingestSegment",
        "dataSource"   : "source-datasource",
        "interval": "2019-10-11T00:00/2019-10-11T02:00"
      },
      "appendToExisting" : false
    },
    "tuningConfig" : {
      "type" : "index_parallel",
      "maxNumSubTasks": 6
    },
    "dataSchema": {
      "dataSource": "destination-datasource",
      "parser": {
        "type": "string",
        "parseSpec": {
          "format": "json",
          "timestampSpec": {
            "column": "STAMP_UPDATED",
            "format": "auto"
          },
          "dimensionsSpec": {
            "dimensions": [
              "NAME",
              "LOCATION",
              "TAG_IN",
              "TAG_OUT",
              "TAG_IN_NAME",
              "TAG_OUT_NAME",
              "TAG"
            ]
          }
        }
      },
      "metricsSpec": [
        { "type": "longSum", "name": "VALUE", "fieldName": "VALUE" }
      ],
      "transformSpec": {
        "transforms": [
          { "type": "expression", "name": "TAG_IN_NAME", "expression": "lookup(TAG_IN, 'tags')" },
          { "type": "expression", "name": "TAG_OUT_NAME", "expression": "lookup(TAG_OUT, 'tags')" },
          { "type": "expression", "name": "TAG", "expression": "array(TAG_IN_NAME, TAG_OUT_NAME)" }
        ]
      },
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "day",
        "queryGranularity": "fifteen_minute",
        "rollup": true
      }
    }
  }
}

Expected behaviour

The resulting records containing an array composed of the values of the specified fields.

NAME | LOCATION | TAG_IN | TAG_OUT | TAG_IN_NAME | TAG_OUT_NAME | TAG
---- | -------- | ------ | ------- | ----------- | ------------ | ---
Foo  | Bar      | 345    | 456     | Super       | Duper        | ["Super", "Duper"]

Actual behaviour

The resulting cells don't contain an array of the actual values but rather values like [Ljava.lang.String;@768e1c0 where the part after the @ changes. This also happens when using kafka indexing tasks as opposed to native batch ingestion.

NAME | LOCATION | TAG_IN | TAG_OUT | TAG_IN_NAME | TAG_OUT_NAME | TAG
---- | -------- | ------ | ------- | ----------- | ------------ | ---
Foo  | Bar      | 345    | 456     | Super       | Duper        | [Ljava.lang.String;@768e1c0

[clintropolis]

Thanks for the issue report. As you've run into, the work done so far in service of #7525 has not yet been fully completed and is not properly wired into the ingestion side through transformation expressions. Apologies for not calling this out in the release notes for 0.16, I will update them now to mention that ingestion side use of multi-value expressions will not function correctly, and try to see if I can get this implemented in time for 0.17.

related #8613.

[fstolba]

Allright, thank you very much for your quick response and the clarification.

Actually, I missed the part of the release notes that state

use of multi-value expressions at ingestion time via a transformSpec will be available in a future release.

However to be absolutely clear I think it might be favorable to add a note to the corresponding section of the Expression docs page as well.

[clintropolis]

Actually, I missed the part of the release notes that state

Hah, no I edited them to add that after responding to this issue 😅.

I agree the docs should also mention this, I will open a PR to add this soon and hopefully get it published sometime in the next day or so.

[denisesenguel]

Running into this issue as well with Druid 0.17.1. when using string_to_array at ingestion time!