build v9 directly

[KurtYoung]

This PR tracks the feature of building v9 directly which had been discussed in https://groups.google.com/forum/#!topic/druid-development/0CxhljSGeeo

We can divide this PR into 3 main parts:

1. Changing of ColumnPartSerde's interface. I had change ColumnPartSerde's interface to separate the Serializer and Deserializer. So we can have multiple serializers, one of it has the same behavior with old ones. And another one is the new streaming version. The Deserializer just do the same thing like old ones.
2. Added IndexedIntsWriter interface and some *Writers which are responsible for writing dimension ids. You can see the hierarchy that i divided the writers to 2 main sub abstract classes: SingleValueIndexedIntsWriter and MultiValueIndexedIntsWriter.
   Here are the classed which are doing the real things:
   VSizeIndexedIntsWriter (single value, vsize encoded, not compressed)
   CompressedIntsIndexedWriter (single value, not vsize encoded, compressed)
   CompressedVSizeIntsIndexedWriter (single value, vsize encoded, compressed)
   VSizeIndexedWriter (multi value, both offset and values are vsized, not compressed)
   CompressedVSizeIndexedV3Writer (multi value, only values are vsized, compressed)
   More details can be found here: https://groups.google.com/forum/#!topic/druid-development/0CxhljSGeeo
3. GenericColumnSerializer and its sub-classes are for writing metrics. They are almost identical to MetricColumnSerializer you familiar with, except the GenericColumnSerializer can write to a Channel(which is allocated from smoosher when building). The sub-classes are:
   LongColumnSerializer (write long metrics)
   FloatColumnSerializer (write float metrics)
   ComplexColumnSerializer (writer complex metrics)

[nishantmonu51]

can we move close to a finally block.

[fjy]

@KurtYoung This is an awesome PR and one that I know several folks want to code review. Everything is going to be slow during the Christmas holidays, but this will get attention after new years.

[nishantmonu51]

👍, seems good to go after a minor nitpick for closing in finally block.
@KurtYoung thanks for the awesome contribution.

[KurtYoung]

added IndexMergerV9, changed some low level interfaces but totally compatible with the old way.
IndexMergerV9 has not been tested, just make the whole thing compiled and old tests passed

Some explanation and thoughts here:

1. Changes to ColumnPartSerde's interface is to separate the Serializer and Deserializer. So we can have multiple serializers, one of it has the same behavior with old ones. And another one is the new streaming version.
2. All writers(time, dim, metrics) are backed by tmp files and copy to the final smoosh file directly.
3. GenericColumnSerializer and its sub-classes are almost identical to MetricColumnSerializer, except the GenericColumnSerializer can write to a Channel(which is allocated from smoosher when building). The MetricColumnSerializer are used for build v8 index, and can be removed later.
4. IndexedIntsWriter are responsible for writing dimension ids which now has 4 types of the formats, more details are here: https://groups.google.com/forum/#!topic/druid-development/0CxhljSGeeo

And here are some points I think should be discussed with your guys when writing the codes:
1. about skippedDimensions in IndexMaker: dimensions with cardinality 0 are treated as skipped dimensions. But in current version of IncrementalIndex, even null dim value are converted to empty string and treated as a valid value, so I assume all the dimensions will be effective so there are no logic about skipped dimensions.
2. about nullSet when building inverted index both in IndexMaker and converting v8 to v9, it checks all dimension's null rows and make sure Dictionary and BitmapIndex are having these null rows. As mentioned in 1, the null values are treated properly in IncrementalIndex, so I did not handle this situation also.

[binlijin]

New two IntBuffer to reuse, when after write switch them to make sure GenericIndexedWriter can sort correct.

[KurtYoung]

hmm...I see the point why the null dict value and null row set are handled both in IndexMaker and IndexIO's converting.

My previous decision about skippedDimension & nullSet are wrong, just ignore it.

Working on this now...

[KurtYoung]

Found a bug of merge & maker about dimension orders:
#2162
I will add some unit tests and marked Ignore, let's just fix it in another PR

[KurtYoung]

Is it ok to insert null to every dimension's dictionary even if the dimension did not contain any null values?
By doing this, there will be no need to iterate all rows just want to determine whether one dimension contains null value and to bump dictionary id if we did not.

Update: Found a way to deal with null value now, but it's a little tricky(there are comments in IncrementalIndexAdapter) and easy to create inconsistency(IncrementalIndexStorageAdapter also rely on IncrementalIndex but does not have this logic right now or it's just does not need this right now). I think proposal above is a possible and easy solution, what do you guys think?

[navis]

Wishfully -1 could be used for null and number of distincts would be specified for each dimension in meta. Would be possible?

[nishantmonu51]

@KurtYoung: are there any corresponding changes in the filters/query path for null handling in case we add null to every dimension dictionary.

[KurtYoung]

@nishantmonu51 I'm also aware of this, the current implementation did not add null to each dimension but handled null value in both IncrementalIndexAdapter and IndexMergerV9.

[fjy]

@KurtYoung there's been a lot of optimizations in the old index merger over the last few months. Are those optimizations incorporated in building the v9 segment directly?

[fjy]

since we are changing the behavior of this method, can we please add a comment on the interface about how the method is supposed to be used?

[fjy]

@KurtYoung I can't find the logic where you actually use index merger v9 instead of index merger

[fjy]

@KurtYoung I think the best way to think about how to handle nulls/empty strings in Druid is described in this PR: #995

[fjy]

I did a first pass over this PR but didn't go into detail for IndexMergerV9. Will look into more once we that know it reasonably works. High level I'm on board with the changes.

[KurtYoung]

@fjy Actually, I did not change any logic to use IndexMergerV9 now, but I switch the current IndexMerger's logic to IndexMergerV9's and make all the test cases passes.
I think maybe it's a good time to switch to IndexMergerV9 after you passed this PR.

[fjy]

@KurtYoung can't seem to make any comments for indexmergerv9

[fjy]

I am not able to make any comments for IndexMergerV9 below this line.

[fjy]

is IndexMergerV9 just a rename of IndexMerger.java?

[fjy]

If that's the case, did you remove the v8 to v9 conversion step in IndexMerger?

[KurtYoung]

IndexMergerV9 make v9 index files directly, the main step are very like with IndexMerger except the v8 to v9 conversion step is no longer necessary.

[fjy]

@himanshug Any more comments?

@KurtYoung Can we add some way to switch between IndexMerger and IndexMerger v9 in the configuration? Index Merger should be the default.

[himanshug]

@fjy I am 👍 once the configuration to switch to IndexMergerV9 is in place.

[xvrl]

I believe @cheddar should have a look at this one. He had a lot of opinions about format when I made changes to introduce dimension compression, and this one introduces even more changes.

I also agree with @gianm we should be able to switch between implementations until it has been verified to be production ready.

[KurtYoung]

@fjy ok, i will add a config about this. Also looking forward to @cheddar 's comments :)

[KurtYoung]

Added "buildV9Directly" option to TuningConfig, docs are updated

[fjy]

@himanshug @xvrl we good to move forward?

[himanshug]

👍 for me

[fjy]

👍 for me too. Will leave this open until tomm to see if anyone else has comments.

@KurtYoung have you filled out the CLA: http://druid.io/community/cla.html

You guys might consider a corporate CLA.

[himanshug]

@fjy @KurtYoung pls squash the commits / cleanup the history, very useful contribution.

[xvrl]

@fjy @KurtYoung I might be wrong, but I still see some comments outstanding. Can we respond or address them?

• https://github.com/druid-io/druid/pull/2138/files#r48401969
• build v9 directly #2138 (comment)
• build v9 directly #2138 (comment)
• build v9 directly #2138 (comment)
• build v9 directly #2138 (comment)

[KurtYoung]

@fjy I have filled out individual CLA, don't know if i had the right to fill a corporate CLA.

[fjy]

@KurtYoung thanks

@xvrl any more comments?

[KurtYoung]

@xvrl All these comments had been addressed and solved.

[KurtYoung]

squashed into 3 commits.

[xvrl]

since buildV9Directly is never null, this should probably be boolean instead of Boolean

[xvrl]

and also renamed to isBuildV9Directly