Don't store all-null dimension columns

[leventov]

When a Druid segment is created, it could be checked that a dictionary-encoded dimension column contains all null values, and just not written to the segment if so.

Probably the same couldn't be done for all-zero numeric columns, because if they are queried as dimensions they should appear as dimensions with all "0" values, not "null" values.

[manishmalhotrawork]

@leventov that would be a good idea.
@gianm do you have any thoughts on that, as that can have impact while searching that dimension?
Plus I can take this task, but needs to dig bit more, as right now getting more familiarized with the code.

[clintropolis]

@leventov do you have something in mind like adding a new 'version' to the dictionary column part header to indicate a constant null dimension if we notice that the dictionary only has null at serialization time, so that way the header of the part is still written, just not the sub parts in order to be able to still distinguish an all null dimension from a dimension that doesn't exist?

With regards to constant elimination (all-zero columns), #6016, does this at the "column chunk scale", by collecting statistics within the chunk in a class called IntFormMetrics which is passed row values as they are sent to the serializer.

These statistics are discarded after the chunk is encoded, but longer term, I was considering saving until all chunks were processed, and making additional optimizations to eliminate constant columns almost entirely by encoding that information in the int column header, allowing the column supplier to return specialized/optimized implementations for these situations. I would imagine all null columns would also be relatively cheap represented in this manner, but would have more overhead in the form of a few function calls, since we could i think we could just push all of this up into a specialized selector that always returns nulls and wouldn't have to call into dictionaries/columns/indexes if we have a specialized dictionary column part for nulls like you are suggesting?

Additionally, I was planning to explore if that same information could be used beyond serialization of the int column to advise on index creation, e.g. whether to use concise or roaring (i need to do some experimentation here to see if any patterns pop up that would be good indicators to decide one way or another), or if we add any additional types of indexes in the future.

I think maybe also would be worthwhile to continue the discussion of #5347 since this relates to segment format perhaps? I plan on weighing in on that issue based on my experiences working on that branch in the very near future.

[leventov]

I don't think this particular thing requires any format change.

in order to be able to still distinguish an all null dimension from a dimension that doesn't exist?

I think you don't need to be able to distinguish them.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue is no longer marked as stale.

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.