[C++] Support reading non-dictionary encoded binary Parquet columns directly as DictionaryArray

[asfimport]

If the goal is to hash this data anyway into a categorical-type array, then it would be better to offer the option to "push down" the hashing into the Parquet read hot path rather than first fully materializing a dense vector of ByteArray values, which could use a lot of memory after decompression

Reporter: Wes McKinney / @wesm
Assignee: Hatem Helal / @hatemhelal

Related issues:

• [C++] Enable reading from ByteArray and FixedLenByteArray decoders directly into arrow::BinaryBuilder or arrow::BinaryDictionaryBuilder (relates to)
• [Python] Support reading Parquet binary/string columns directly as DictionaryArray (is related to)

PRs and other links:

• GitHub Pull Request #3721

_{Note: This issue was originally created as ARROW-3769. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Moved this here from the Parquet JIRA

[asfimport]

Wes McKinney / @wesm:
This is implemented in in #3492 for PARQUET-1508, but not tested. This JIRA should be used to add unit tests and probably some benchmarks, too

[asfimport]

Hatem Helal / @hatemhelal:
I've started looking into this and starting with some unit tests to make sure I understand the inner workings.  

[asfimport]

Wes McKinney / @wesm:
Cool. This is only implemented at the encoder level, so you should be able to use ArrayFromJSON to make writing the unit tests easier – so for this JIRA I am expecting tests in parquet-encoding-test

[asfimport]

Hatem Helal / @hatemhelal:
Made a start on the unittests here:

mathworks#12

@wesm, could you take a look and let me know if this is heading in the right direction?

[asfimport]

Wes McKinney / @wesm:
yes, I think that's the basic idea

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 3721
#3721