Indexer does not merge properly when dimensions are provided in non-lexicographic order

[gianm]

If you give the indexer dimensions in some arbitrary order, the IncrementalIndex will sort its rows based on the values of dimensions in that order. Then, when the IndexMerger merges multiple indexes, it uses a MergeIterable that assumes rows are sorted on values of dimensions in lexicographic order instead of the order the underlying segments actually use. This causes the rowboats to come in out of order and not get combined properly, and the resulting segments have more rows than they should.

Workarounds include: Supply dimensions in lexicographic order in the indexer config. Avoid indexing with dimensionless schemas.

I can think of two possible fixes:

• IndexMerger could try to use arbitrary dimension orders from the underlying segment when possible. I think this is not possible in general while both getting correct rollup and avoiding re-sorting rows, since the underlying segments could use different row comparators. But it will be possible in many situations. The IndexMerger can also decide to sort rows in situations where it would need to.
• IncrementalIndex could sort rows based on dimensions in lexicographic order instead of arbitrary order. This makes the IndexMerger's job easier but it means we can't control the row comparator, which could be useful if we ever decide to mess with it in order to do something like optimize size of the inverted indices.

[srini-mp]

Does arbitrary order make sense, if you want your commonly queried dimensions to be sorted next to each other, indexed and merged together ?

[cheddar]

@srini-mp If you are specifying the dimensions, then no, this is not a problem. It becomes a problem when you are ingesting data in a schema-less fashion and the sets of dimensions seen don't always happen in the same order.

[himanshug]

@gianm @cheddar I'm trying to understand this one and looking at the code. though I could understand the problem and both proposed solutions. but, I am having hard time understanding the problem with implementing option-2 that is "optimize size of inverted indices" . can u pls explain this a bit more?

[gianm]

@himanshug I think nobody actually does this, but, in principle you could choose a dimension ordering that creates more runs in the earlier columns and that would make compression work better for those columns.

[drcrallen]

For example, if you have some low cardinality columns, you can get their bitmaps to compress really well by grouping them together. High cardinality columns should have pretty sparse bitmaps anyways, so I wouldn't expect them to benefit much from grouping together.

[drcrallen]

druid-io/druid-api#68 removed a workaround for this so I'm tagging it with 0.9.0

[fjy]

@gianm can we close this?