Quantiles sketch agg fails on inner query numeric post-agg columns

[jon-wei]

Affected Version

0.12.0+

Description

When using a quantiles sketch agg (http://druid.io/docs/latest/development/extensions-core/datasketches-quantiles.html) in the outer query of a nested GroupBy that references a numeric column generated by a post-agg in the inner query, the following exception occurs:

java.lang.ClassCastException: java.lang.Double cannot be cast to com.yahoo.sketches.quantiles.DoublesSketch
	at org.apache.druid.query.aggregation.datasketches.quantiles.DoublesSketchMergeBufferAggregator.aggregate(DoublesSketchMergeBufferAggregator.java:65) ~[?:?]
	at org.apache.druid.query.groupby.epinephelinae.AbstractBufferHashGrouper.aggregate(AbstractBufferHashGrouper.java:165) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.groupby.epinephelinae.SpillingGrouper.aggregate(SpillingGrouper.java:167) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.groupby.epinephelinae.Grouper.aggregate(Grouper.java:82) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$1.accumulate(RowBasedGrouperHelper.java:270) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$1.accumulate(RowBasedGrouperHelper.java:247) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]

This occurs because the factorizeBuffered method in DoublesSketchAggregatorFactory relies on metricFactory.getColumnCapabilities(fieldName) to determine if an input column is numeric. If the column is not numeric, the aggregator assumes the input is a complex DoublesSketch object. For postaggs, the type information is not available, so the type mismatch occurs.

This issue may also be present in other aggregator types, I have not searched through the other implementations.

The following query structure will reproduce the issue:

{
  "queryType": "groupBy",
  "intervals": [
    "2015-09-12/2015-09-13"
  ],
  "dataSource": {
    "type": "query",
    "query": {
      "queryType": "groupBy",
      "dataSource": "wikipedia",
      "intervals": [
        "2015-09-12/2015-09-13"
      ],
      "dimensions": [
        "page"
      ],
      "aggregations": [
        {
          "type": "quantilesDoublesSketch",
          "name": "innerSketch",
          "fieldName": "added"
        },
        {
          "type": "count",
          "name": "sampleCount"
        }
      ],
      "postAggregations": [
        {
          "type": "quantilesDoublesSketchToQuantile",
          "name": "innerMedian",
          "field": {
            "type": "fieldAccess",
            "fieldName": "innerSketch"
          },
          "fraction": 0.5
        }
      ],
      "granularity": "all"
    }
  },
  "dimensions": [
    "page"
  ],
  "aggregations": [
    {
      "type": "quantilesDoublesSketch",
      "name": "outerSketch",
      "fieldName": "innerMedian"
    },
    {
      "type": "count",
      "name": "clientCount"
    }
  ],
  "postAggregations": [
    {
      "type": "quantilesDoublesSketchToQuantile",
      "name": "outerMedian",
      "field": {
        "type": "fieldAccess",
        "fieldName": "outerSketch"
      },
      "fraction": 0.5
    }
  ],
  "granularity": "all",
  "context": {
    "skipEmptyBuckets": "true"
  }
}

[leerho]

@jon-wei
We are looking at this. We think it is a problem in the aggregator and not the sketch itself.

[jon-wei]

@leerho

Thanks! The issue is with the aggregator (or perhaps with Druid's column capabilities not providing info about post-aggs), a quick fix would be to allow the user to choose and force the use of the build/merge aggregator in the AggregatorFactory properties.

A larger adjustment (non-aggregator specific) that would also fix this would be for Druid to provide type information about postagg columns in the capabilities.

[clintropolis]

A larger adjustment (non-aggregator specific) that would also fix this would be for Druid to provide type information about postagg columns in the capabilities.

FYI, I'm investigating this approach due to necessity for a more optimized handling of group by stuff from row selector factories for #7588.

[leerho]

@jon-wei @clintropolis
It is clear to us that you folks have a much deeper understanding of the Druid infrastructure and what is required to fix the aggregator. Is it fair to assume you folks will take the lead on this? If there are any issues wrt the sketch itself we would gladly help.

[clintropolis]

It is clear to us that you folks have a much deeper understanding of the Druid infrastructure and what is required to fix the aggregator. Is it fair to assume you folks will take the lead on this? If there are any issues wrt the sketch itself we would gladly help.

Sure, I'll update this ticket if my approach doesn't pan out or runs into any issues specific to this aggregator

[gianm]

The HllSketch aggregator seems to have a similar issue.

[himanshug]

Theta Sketch[Buffer]Aggregator just does type check and dispatch, which would possibly be the quickest fix.

[clintropolis]

Theta Sketch[Buffer]Aggregator just does type check and dispatch, which would possibly be the quickest fix.

In the interest of sooner than later, I think I'm going to add this approach as a fallback for when column capabilities are not available for a column, because it is definitely the quickest fix. Typing all the things is a bit more involved, and I still plan on doing because it would offer improvements here allowing us to take optimized paths when types are known up front, but it can be done at a more leisurely pace.