[Rust] [DataFusion] Add better and faster support for dictionary types

[asfimport]

Usecases: Efficiently process large columns of low cardinality Strings
 

• BatchIterator should accept both DictionaryBatch and RecordBatch
• Type Coercion optimizer rule should inject expression for converting dictionary value types to index types (for equality expressions, and IN(values, ...)
• Physical expression would lookup index for dictionary values referenced in the query so that at runtime, only indices are being compared per batch

Reporter: Andy Grove / @andygrove

Related issues:

• [Rust][DataFusion] Add support for Dictionary types in data fusion (relates to)

_{Note: This issue was originally created as ARROW-8464. Please see the migration documentation for further details.}

[asfimport]

Andrew Lamb / @alamb:
FYI @andygrove – I am doing some part of this in ARROW-10159 – however, the initial implementation effectively converts DictionaryArray --> PrimitiveArray / StringArray and then uses the existing processing.

To support the actual efficient processing usecase, I think significant work will be needed to add appropriate dictionary support to the arrow compute kernels

[asfimport]

Andrew Lamb / @alamb:
Migrated to github: apache/datafusion#87