TransformSpec for metrics when re-indexing a datasource using ingestSegment appears to be broken

[suneet-s]

Affected Version

0.18

Description

The following re-index spec spits out 0s for both "double-deleted" a transformation on a column "deleted" and "one-plus-triple-deleted" triple-deleted is a metric calculated from the original index job's transformSpec

{
    "type": "index",
    "spec": {
        "dataSchema": {
            "dataSource": "reindex-double-tfn-1",
            "metricsSpec": [
                {
                    "type": "doubleSum",
                    "name": "added",
                    "fieldName": "added"
                },
                {
                    "type": "doubleSum",
                    "name": "triple-added",
                    "fieldName": "triple-added"
                },
                {
                    "type": "doubleSum",
                    "name": "one-plus-triple-added",
                    "fieldName": "one-plus-triple-added"
                },
                {
                    "type": "doubleSum",
                    "name": "deleted",
                    "fieldName": "deleted"
                },
                {
                    "type": "doubleSum",
                    "name": "double-deleted",
                    "fieldName": "double-deleted"
                },
                {
                    "type": "doubleSum",
                    "name": "delta",
                    "fieldName": "delta"
                }
            ],
            "granularitySpec": {
                "segmentGranularity": "DAY",
                "queryGranularity": "second",
                "intervals" : [ "2013-08-31/2013-09-01" ]
            },
            "parser": {
                "parseSpec": {
                    "format" : "json",
                    "timestampSpec": {
                        "column": "timestamp",
                        "format": "iso"
                    },
                    "dimensionsSpec": {
                        "dimensions": [
                            "page",
                            {"type": "string", "name": "language", "createBitmapIndex": false},
                            "user",
                            "unpatrolled",
                            "newPage",
                            "anonymous",
                            "namespace",
                            "country",
                            "region",
                            "city"
                        ]
                    },
                    "transformSpec": {
                        "transforms": [
                            {
                                "type": "expression",
                                "name": "newPage",
                                "expression": "page"
                            },
                            {
                                "type": "expression",
                                "name": "one-plus-triple-added",
                                "expression": "triple-added + 1"
                            },
                            {
                                "type": "expression",
                                "name": "double-deleted",
                                "expression": "deleted * 2"
                            }
                        ]
                    }
                }
            }
        },
        "ioConfig": {
            "type": "index",
            "firehose": {
                "type": "ingestSegment",
                "dataSource": "wiki-tests-2",
                "interval": "2013-08-31/2013-09-01"
            }
        },
        "tuningConfig": {
            "type": "index"
        }
    }
}

The ingest spec for wiki-tests-2 is the same as the one described in #9589

[suneet-s]

This works if you use a DruidInputSource instead of the ingestSegmentFirehose

[2bethere]

What's the expected output?

[abhishekagarwal87]

@suneet-s It seems that ParserSpec does not have the transformSpec field and that's why is ignoring it.

 @JsonCreator
  public JSONParseSpec(
      @JsonProperty("timestampSpec") TimestampSpec timestampSpec,
      @JsonProperty("dimensionsSpec") DimensionsSpec dimensionsSpec,
      @JsonProperty("flattenSpec") JSONPathSpec flattenSpec,
      @JsonProperty("featureSpec") Map<String, Boolean> featureSpec,
      @JsonProperty("keepNullColumns") Boolean keepNullColumns
  )

That is why transform expressions are getting ignored.

[abhishekagarwal87]

There is likely another bug though. When transformSpec is moved from Parser to dataSchema, the transformations are applied twice.
Once in InputSourceProcessor

final InputSourceReader inputSourceReader = dataSchema.getTransformSpec().decorate(
        inputSource.reader(
            new InputRowSchema(
                dataSchema.getTimestampSpec(),
                dataSchema.getDimensionsSpec(),
                metricsNames
            ),
            inputFormat,
            tmpDir
        )
    );

and other in DataSchema#getParser()

inputRowParser = transformSpec.decorate(objectMapper.convertValue(this.parserMap, InputRowParser.class));

Test query fails if I move the transformSpec out
ExpectedOutput:

{
  "version": "v1",
  "timestamp": "2013-08-31T00:00:00.000Z",
  "event": {
    "added_count_times_ten": 27160,
    "newPage": "Crimson Typhoon",
    "city": "city-Taiyuan",
    "double_deleted_count": 10,
    "delta_overshadowed": 450,
    "added_count": 2716
  }
}

Actual output

{
  "version": "v1",
  "timestamp": "2013-08-31T00:00:00.000Z",
  "event": {
    "double_deleted_count": 10,
    "delta_overshadowed": 225,
    "city": "city-city-Taiyuan",
    "newPage": "Crimson Typhoon",
    "added_count_times_ten": 27160,
    "added_count": 2716
  }
}

[suneet-s]

@abhishekagarwal87 Thanks for digging in to this! It looks like I didn't follow the instructions on using a transformSpec correctly... So the behavior described in this issue is correct.

I'm going to close this issue, since the issue you found of a transformSpec being applied twice is described in #9589