implement special distinctcount

[binlijin]

Discuss at https://groups.google.com/forum/#!topic/druid-development/GXRpXfBzfJs
(1) First use #2570 to partition data by visitor_id.
(2) Second use distinctcount to calculate exact UV.
There is some limitations, when use with groupBy, the groupBy keys' numbers should not exceed maxIntermediateRows in every segment, if exceed the result will wrong. And when use with topN, numValuesPerPass should not too big, if too big the distinctcount will use many memory and cause the JVM out of service.

[fjy]

@binlijin how fast is the exact distinct count?

[fjy]

@binlijin can you add a README to the module? Something to help explain how to do the distinct count and also what to watch out for. If you guys use this in production, can you also note it is used in production in the README?

[binlijin]

@fjy, the exact distinct count is faster than thetaSketch ( which use size 16384).
We have done performance test in our perf cluster, and have not use in production, will use this in production lately when we done all the test and prepare all the data.
This PR is for pre review if any one interesting in it.

[fjy]

@binlijin there's some merge conflicts

[fjy]

@binlijin we'll need some docs on how to use this

[fjy]

let's put this in extensions-contrib for now

[binlijin]

done

[binlijin]

Performance:
Sketch query

{
  "queryType":"timeseries",
  "dataSource":"someDataSource",
  "granularity":"day",
  "intervals":["2016-03-06T00:00:00/2016-03-06T23:59:59"],
  "aggregations":[
    {
      "fieldName":"visitor_id_sketch",
      "name":"uv",
      "type":"thetaSketch"
    }
  ],
  "context" :  {
    "useCache":"false",
    "populateCache":"false"
  }
}

Take time:

real    0m3.286s
user    0m0.001s
sys 0m0.002s

[binlijin]

DistinctCount query:
Use java bitmap

{
  "queryType":"timeseries",
  "dataSource":"someDataSource",
  "granularity":"day",
  "intervals":["2016-03-06T00:00:00/2016-03-06T23:59:59"],
  "aggregations":[
    {
      "fieldName":"visitor_id",
      "name":"uv",
      "bitmap": {"type" : "java"},
      "type":"distinctCount"
    }
  ],
  "context" :  {
    "useCache":"false",
    "populateCache":"false"
  }
}

Take time:

real    0m0.973s
user    0m0.001s
sys 0m0.002s

[binlijin]

DistinctCount query:
Use roaring bitmap
"bitmap": {"type" : "roaring"},
Take time:

real    0m2.348s
user    0m0.001s
sys 0m0.002s

[binlijin]

DistinctCount query:
Use concise bitmap
"bitmap": {"type" : "concise"},
Take time:

real    3m40.176s
user    0m0.000s
sys 0m0.003s

[himanshug]

instead of 2 links above can you please document the process of how this extension could be used.

[himanshug]

also pls indicate the assumption that one value has to be present in only one segment or this might overcount.

[binlijin]

ok

[himanshug]

i think this might need to encdoe the name and bitmap type too.

[binlijin]

ok

[fjy]

@binlijin

Do you mind adding an entry for this new extension under
docs/content/development/extensions.md

and instead of the README.md, create a new file under
docs/content/development/extensions-contrib

?

This will make it much easier to version extensions and list them on the Druid webpage

[fjy]

@binlijin also please squash commits

@himanshug any more comments?

[fjy]

Also, wait for #2698 to be merged before making doc changes :)

[binlijin]

@fjy, ok, will change it and squash commits, after #2698 merge.

[fjy]

@binlijin merged #2698

[himanshug]

you might want to take care of the case where one of the segment does not have fieldName column
see https://github.com/druid-io/druid/blob/master/extensions-core/datasketches/src/main/java/io/druid/query/aggregation/datasketches/theta/SketchAggregatorFactory.java#L77

[binlijin]

done, thanks

[himanshug]

👍 after #2602 (comment) and #2602 (comment) are resolved and commits are squashed.

[binlijin]

@fjy i find the link in docs/content/development/extensions.md is wrong, for example:
"|druid-rocketmq|RocketMQ firehose.| [link] (../development/extensions-contrib/rocketmq.html)|"
there is not rocketmq.html in ../development/extensions-contrib/, only rocketmq.md, should fire another PR to fix this.

[fjy]

@binlijin the html file is generated when teh docs are ported over to the webpage, you shouldn't need to fix anything

[fjy]

@binlijin for example, under docs/content, you can run jekyll serve to render the docs and the html page should be rendered

[binlijin]

@fjy, ok, so i need to minor change this patch.

[binlijin]

@fjy @himanshug i think all comments have been resolved.

[himanshug]

👍 for me

[drcrallen]

I don't think this line is needed.

[fjy]

i think it is useful to know this

[drcrallen]

Approach looks good.

If you are wanting JUST counts, this can be sped up by implementing this functionality at a query level, and simply returning the cardinality of the resulting filter.

[drcrallen]

Strongly suggest fixing #2602 (comment) but the rest is at the author's discretion. 👍

[leventov]

@binlijin implementation of DistinctCountAggregatorFactory.combine() doesn't really combine distinct counts

[PayneZx]

@binlijin When I used distinctCount aggregation type, I find some bugs.For example,when dimensions size is two,the result will be wrong.I don't know if it is a bug.