More specific null/empty str handling in IndexMerger

[jon-wei]

Addresses a concern raised in PR #2006 about dimension cardinalities.

Changes the conditions for when an entry for the null/empty str is added to the value dictionary during index merging.

Old behavior:

• If dim has values in at least one index, ensure the null/empty str is in the dictionary and convert missing columns to empty strings for merging

New behavior:

• If dim is entirely missing from an index, but has values in another index, ensure the null/empty str is in the dictionary and convert missing columns to empty strings for merging

[fjy]

can we have a more descriptive name?

[jon-wei]

@fjy changed this to "dimAbsentFromIndex"

[fjy]

@navis does this address your concerns from #2006 ?

[fjy]

👍

[binlijin]

What will happen if a dim has values in all index, but all index do not have null/empty value?

[jon-wei]

@binlijin The null/empty str will not be added to the dictionary in that case (i.e., dimAbsentFromIndex will be false)

Off the top of my head, testNonLexicographicDimOrderMerge() in IndexMergerTest is an example where the dims don't have null/empty values

[binlijin]

@jon-wei good

[gianm]

this should be &&

[jon-wei]

@gianm changed, thanks for catching that

[gianm]

@jon-wei It seems like the ideal condition would be:

• If dim is entirely missing from one index, but has mixed nulls and values in another index, ensure the null/empty str is in the dictionary and convert missing columns to empty strings for merging

Checking for mixed nulls/values shouldn't add additional expense as isNullColumn already iterates over all the dim values. So I think it's feasible to do this check the ideal way.

[jon-wei]

@gianm It's still necessary to add the null/empty str to the dictionary for the case where a dim is missing from one index, and has no null values in another (missing and non-mixed) when they merge.

In the legacy merger this was handled by bumping the dictionary in IndexIO:

          if (onlyOneValue) {
            log.info("Dimension[%s] is single value, converting...", dimension);
            final boolean bumpedDictionary;
            if (nullsSet != null) {
              log.info("Dimension[%s] has null rows.", dimension);
              final ImmutableBitmap theNullSet = bitmapFactory.makeImmutableBitmap(nullsSet);

              if (dictionary.get(0) != null) {
                log.info("Dimension[%s] has no null value in the dictionary, expanding...", dimension);
                bumpedDictionary = true;
                final List<String> nullList = Lists.newArrayList();
                nullList.add(null);

                dictionary = GenericIndexed.fromIterable(
                    Iterables.concat(nullList, dictionary),
                    GenericIndexed.STRING_STRATEGY
                );

[gianm]

👍