Single dimension hash-based partitioning

[binlijin]

Dimension hash-based partitioning.
Partitioning rows across those segments according to the hash of the partition dimension in each row. So all rows with a particular value for that dimension will go into the same segment.

See the requirement https://groups.google.com/forum/#!topic/druid-user/yfAzwStIZGo
and https://groups.google.com/forum/#!topic/druid-development/GXRpXfBzfJs

[b-slim]

@binlijin can you please

1. add documentation
2. add UTs at least ser deser
3. explain how this sort of solve the two issues you have mentioned above ?

Thanks !

[binlijin]

@b-slim , ok, i will add these.

[xvrl]

@binlijin I'm wondering if instead of adding a new type of partitioning we could extend the existing "hashed" type to support an arbitrary number of dimensions. I think it would also help reduce the confusion around the different partitioning schemes.

[binlijin]

@xvrl , ok , i will change it lately.

[binlijin]

@b-slim @xvrl @fjy what about now?
We need this to rebuild our product data, and it it important for us, because the ShardSpec information store with segment meta in mysql.

[b-slim]

why this is ignored isn't part of the state ?

[binlijin]

done, have change this to JsonProperty

[b-slim]

can you add @Nullable as method signature

[binlijin]

done

[nishantmonu51]

minor nit: would it make sense and make code more clean to make partitonDimensions an empty list when user does not specify any dimensions ?

[binlijin]

@nishantmonu51 ok, will remove @nullable also, because there is no necessary.

[b-slim]

LGTM after taking care of minor comments.

[fjy]

👍 after my comment on docs is answered

[nishantmonu51]

How would you feel about making this passed as a json property in HashedPartitionsSpec and add it to the docs, so that people can use it ?

[binlijin]

ok

[himanshug]

so, now, what is the difference between following configurations?

{
  "type": "hashed",
  "numShards": n,
  "partitionDimensions": <whatever>
}

and

{
  "type": "dimension",
  "numShards": n,
  "partitionDimensions": <whatever>
}

I think they will produce exactly same results. In other words, I don't think "dimension" partition spec needs to support "numShards" as it is only used in the case when you want to partition data by dimension value range. Fixed shards case is already supported by "hashed" partition spec.

[binlijin]

Yes, they will produce exactly same results, i will remove "dimension" partition spec's "numShards".

[drcrallen]

why doesn't this return "partitionDimension"?

[drcrallen]

With this change should single partition hash spec be deprecated?

[binlijin]

@drcrallen, if you want to partition data by one dimension or dimensions with "numShards", use

{
  "type": "hashed",
  "numShards": n,
  "partitionDimensions": <whatever>
}

If you want to partition data by one dimension with "targetPartitionSize", use

  "partitionsSpec": {
     "type": "dimension",
     "targetPartitionSize": 5000000
   }