IndexMerger & IndexMaker make the wrong result if indexes have different dimension order

[KurtYoung]

The main idea is like IndexMergerTest::testNonLexicographicDimOrderMerge()
If we have 2 indexes:
one have dimension order like "d3", "d1", "d2"
the other have dimension order like "d2", "d1", "d3"
Note in IncrementalIndex, the "facts" are sorted based on its own dimension order.
But when we merge these two indexes, we must determine the merged dimension order, which we can not found one to make these two indexes happy.
So when we merge rows from these two indexes, even if we have changed the dimension order to the new merged order, but the order inside each index are different with the outer merge order.

[fjy]

@KurtYoung what version are you encountering this? And are you absolutely sure the results are incorrect? I know before when we examined this there were more rows than there needed to be, but the results were correct. There have been numerous issues and pull requests related to this feature in the last little while.

#658
#2006
#1974

[KurtYoung]

@fjy I'm reproducing this in master branch and I'm pretty sure the results are incorrect.
Just check out the unit test I added for this issue in this PR:#2138
IndexMergerTest::testConflictDimOrderMerge()
Update: When I say the results are incorrect earlier, I mean the row number is wrong, the content is correct though.

[fjy]

@KurtYoung I wonder if it is the recent pull requests that got merged that are causing this issue. Do you see the problem in 0.8.3?

[fjy]

@jon-wei can you take a look at this issue?

[gianm]

@KurtYoung can you please describe exactly what you mean by "wrong"?

The unit test in #2138 looks like it's detecting rows that are not fully merged (it's getting 6 rows instead of 3) so I'm wondering if you're seeing things that are truly wrong (i.e. lead to wrong query results) or just things that are not fully merged at indexing time.

It's expected that IndexMerger will not merge completely if the dim orders of the two indexes don't match. So there might be two separate rows in the output segment that have the same dimensions and truncated timestamp, and weren't merged at indexing time like they should have been. But, the contents of the rows should still be correct and query results should still be correct. If you're seeing this, that's a known problem (#658) and is meant to be addressed by #2006.

If you're seeing something else then that is bad.

[KurtYoung]

@gianm Yes, I mean the row number is wrong, but the contents are all correct. If this is an expected behavior, then just ignore this issue.

[gianm]

@KurtYoung It's definitely "expected" but not ideal. #2006 is meant to address it for the most common use case of IndexMerger (merging indexes as part of the process of creating segments from raw data). If you have any thoughts on that we would love to hear them.

[KurtYoung]

@gianm IMO why not sort dimension by their names before we persist index and this can save lots of work when we merging these indexes.

[gianm]

@KurtYoung Mostly to support schemaless indexing, where the dimensions are not known ahead of time, and also because letting people customize dimension order can help with compression.

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.