Allow IncrementalIndex to store Long/Float dimensions

[jon-wei]

This PR updates IncrementalIndex such that it can ingest and store Long/Float dimensions in addition to Strings. It is intended as a step towards the larger goals being discussed in the following topic:

https://groups.google.com/forum/#!topic/druid-development/obtfNJnXPDg/discussion

• DimDim is now a generic class instead of assuming String values. The TimeAndDims grouping key class now stores dimension values as Comparable[] instead of String[].
• The PR does not change the DimensionSelector or *Adapter interfaces to support non-String types, so query/index persist behavior is unchanged, nor does this PR provide a way for the user to specify non-String dimension types in the ingestion schema.
• Schema-less dimension type discovery is not enabled in the PR (newly discovered dims are assumed to be Strings).
• The PR includes a JMH benchmark that adds rows with Long/Float/String columns to an IncrementalIndex.

I ran the row-adding benchmark with type discovery enabled for testing purposes and the numbers are shown below:

java -jar target/benchmarks.jar IncrementalIndexAddRowsBenchmark -f 1 -i 20 -wi 10

With patch, non-String dims:

Run complete. Total time: 00:08:05

Benchmark                                       Mode  Cnt   Score   Error  Units
IncrementalIndexAddRowsBenchmark.normalFloats   avgt   20  18.918 ± 3.063  us/op
IncrementalIndexAddRowsBenchmark.normalLongs    avgt   20  10.783 ± 0.394  us/op
IncrementalIndexAddRowsBenchmark.normalStrings  avgt   20  28.218 ± 2.513  us/op

Without patch, String dims only:

Run complete. Total time: 00:09:34

Benchmark                                       Mode  Cnt   Score   Error  Units
IncrementalIndexAddRowsBenchmark.normalFloats   avgt   20  19.977 ± 2.803  us/op
IncrementalIndexAddRowsBenchmark.normalLongs    avgt   20  20.520 ± 3.298  us/op
IncrementalIndexAddRowsBenchmark.normalStrings  avgt   20  29.656 ± 3.261  us/op

[nishantmonu51]

static variable name in UPPERCASE.

[jon-wei]

@nishantmonu51 addressed

[pjain1]

Just curious, do we have any parser in Druid currently that returns [] type for multi-valued dimension ?

[jon-wei]

@pjain1 Do you mean if the multi-value array can have subitems with an array[] type? If so, I don't think that will occur, the values would be single numerics or Strings for the JSON parsers and all single strings for the delimited parsers, I believe.

[pjain1]

then why there is a need for [] types here as TYPE_MAP is used like TYPE_MAP.get(singleVal.getClass().getSimpleName()) where singleValue as you said "would be single numerics or Strings for the JSON parsers and all single strings for the delimited parsers, I believe."

[jon-wei]

@pjain1 It's not strictly necessary now, I think I had it in there from an older implementation where I used the TYPE_MAP to get a ValueType from a Comparable[].

I kept it there in case, but I can remove it if needed.

[pjain1]

@jon-wei I was just curious to know why it was there. No need to remove if it does not hurt anything.

[jon-wei]

closed for now to rebase to master, will reopen

[drcrallen]

dont' force-push your branch until the PR is open or it will render this unable to re-open

[jon-wei]

@drcrallen yeah, forgot about that yesterday =\

I was able to re-open it though by doing a "git reset --hard" to the SHA at the time the PR was closed

[fjy]

@jon-wei we need to rebase this

[jon-wei]

Updated this to rebase on top of changes introduced by:

#2085

The PR above changed TimeAndDims to use a dictionary encoded int[] array instead of String[]; this patch uses dictionary encoding for Long/Float values as well (unnecessary).

I plan to change TimeAndDims to use Comparable[] instead of int[] in a later PR, still WIP in my fork:
https://github.com/jon-wei/druid/tree/flex_dims_pt2

Strings will store a dictionary encoding in TimeAndDims as in PR #2085, numerics will store their values directly in TimeAndDims.

[fjy]

header

[jon-wei]

@fjy added license header

[fjy]

👍

[nishantmonu51]

please file an issue for this todo.

[nishantmonu51]

What is the behavior of null for Long/Float dimensions ?
Will it throw parseException or successfully ingest null values and treat them as null ?

can we also add tests for null values in Long/Float dimension to make sure it behaves as expected.

[nishantmonu51]

Also need to add docs for the added functionality.

[jon-wei]

@nishantmonu51

RE: null handling, the nulls are being converted to 0. It's not in this PR, but a follow on PR that I am finishing up:

https://github.com/jon-wei/druid/blob/5bffacec5772718bcb948b81e6ace4192d940396/processing/src/main/java/io/druid/segment/IndexMerger.java#L1432

That PR reuses the existing single-value long/float column formats for the numeric dims, representing null without changes to the storage format (need some mechanism to distinguish null from normal values) or using dictionary encoding (have a dictionary entry for null) seems problematic. Throwing an exception sounds okay to me too (maybe let user configure? choice of "throw exception" or "convert null to 0")

RE: docs and tests, this open PR doesn't actually enable the use of long/float dimensions, it's just preparing IncrementalIndex for later changes:

• It doesn't provide a way to specify dimension typing in the schema
• No query/filter/extraction fn, etc. support for long/floats yet

I address those points and have tests in the follow-on patch (not ready yet), I think it would make more sense to include docs in the later PR when the numeric dims are actually usable

[jon-wei]

@fjy @nishantmonu51 will file issues for TODOs

[nishantmonu51]

+1, I think this is ready to be merged. docs can be added in the follow up PR.

[fjy]

@jon-wei do you mind resolving merge conflicts?

[jon-wei]

@fjy rebased