ARROW-3769: [C++] Add support for reading non-dictionary encoded binary Parquet columns directly as DictionaryArray#3721

Closed

hatemhelal wants to merge 35 commits intoapache:masterfrom

mathworks:arrow-3769

Commits on Mar 17, 2019

Add some basic unittests that exercise the DecodeArrow methods

Hatem Helal
authored and
wesm
committed
Implement DecodeArrowNonNull and unit tests

Hatem Helal
authored and
wesm
committed
remove todo message

Hatem Helal
authored and
wesm
committed
added tests for DictByteArrayDecoder and reworked previous tests

Hatem Helal
authored and
wesm
committed
added benchmarks for decoding plain encoded data using arrow builders

Hatem Helal
authored and
wesm
committed
rework data generation so that decoding benchmark runs using a more realistic dataset

Hatem Helal
authored and
wesm
committed
fix appveyor windows failure

Hatem Helal
authored and
wesm
committed
fix loop increment in templated PlainByteArrayDecoder::DecodeArrow method

Hatem Helal
authored and
wesm
committed
respond to code review feedback - many readability fixes in benchmark and tests

Hatem Helal
authored and
wesm
committed
prefer range-based for loop to KeepRunning while loop pattern

Hatem Helal
authored and
wesm
committed
Use value parameterization in decoding tests

Hatem Helal
authored and
wesm
committed
prefer mersenne twister prng over default one which is implemenation defined

hatemhelal
authored and
wesm
committed
prefer default_random_engine to avoid potential slowdown with Mersenne Twister prng

Hatem Helal
authored and
wesm
committed
Add overloads for decoding using a StringDictionaryBuilder

hatemhelal
authored and
wesm
committed
Add benchmark for dictionary decoding using arrow builder

hatemhelal
authored and
wesm
committed
Add support for requesting a parquet column be read as a DictionaryArray

hatemhelal
authored and
wesm
committed
Make sure to update the schema when reading a column as a DictionaryArray

hatemhelal
authored and
wesm
committed
Basic unittests for reading DictionaryArray directly from parquet

hatemhelal
authored and
wesm
committed
Move function definition to (hopefully) resolve appveyor build failure due to C2491

hatemhelal
authored and
wesm
committed
Reworked encoding benchmark to reduce code duplication

hatemhelal
authored and
wesm
committed
remove unnecessary inlines

Hatem Helal
authored and
wesm
committed
replace deprecated ReadableFileInterface with RandomAccessFile

Hatem Helal
authored and
wesm
committed
simplify ArrowReaderProperties and mark as experimental

Hatem Helal
authored and
wesm
committed
Refactor dictionary data generation into RandomArrayGenerator

hatemhelal
authored and
wesm
committed
Rework parquet encoding tests

Hatem Helal
authored and
wesm
committed
Fix DecodeArrow from plain encoded columns

Hatem Helal
authored and
wesm
committed
Temporarily disable tests for arrow builder decoding from dictionary encoded col

Hatem Helal
authored and
wesm
committed
Fix DictEncoding test: need to use PutSpaced instead of Put in setup

hatemhelal
authored and
wesm
committed
Use random string generator instead of poor JSON

hatemhelal
authored and
wesm
committed
revert incorrect changes to PlainByteArrayDecoder::DecodeArrow method

Hatem Helal
authored and
wesm
committed
use PutSpaced in test setup to correctly initialize encoded data

Hatem Helal
authored and
wesm
committed
Rework ByteArrayDecoder interface to reduce code duplication

Hatem Helal
authored and
wesm
committed
Removed dependencies on arrow builder in parquet/encoding

Hatem Helal
authored and
wesm
committed
Add virtual destructor to WrappedBuilderInterface

Hatem Helal
authored and
wesm
committed
Move schema fix logic to post-processing step

hatemhelal
authored and
wesm
committed