Optimized spark vectorized read parquet decimal #3249

ConeyLiu · 2021-10-08T11:52:02Z

Arrow use 16 bytes for all decimal vector, however, the data could be stored as int/long in parquet file for different precision decimal data. We only need to use the int/long arrow vector for int/long backed decimal data. This could improve performance a lot when we do a full table scan on the store_sales table(1TB data scale).

The existed UT should cover the vectorized read case. I could add the extra UTs if needed.

ConeyLiu · 2021-10-08T11:53:00Z

Hi @rdblue @jackye1995 @nastra, could you help to review this? thanks a lot.

nastra · 2021-10-08T12:05:30Z

@ConeyLiu I'll try to review this next week. In the meantime could you please run some benchmarks for this PR and attach the results here?

So in your case it would be: ./gradlew :iceberg-spark3:jmh -PjmhIncludeRegex=VectorizedReadFlatParquetDataBenchmark -PjmhOutputPath=benchmark/vectorized-read-flat-parquet-data-result.txt

ConeyLiu · 2021-10-09T08:46:18Z

before

Benchmark                                                                 Mode  Cnt   Score   Error  Units
VectorizedReadFlatParquetDataBenchmark.readDatesIcebergVectorized5k         ss    5   1.362 ± 0.092   s/op
VectorizedReadFlatParquetDataBenchmark.readDatesSparkVectorized5k           ss    5   1.451 ± 0.222   s/op
VectorizedReadFlatParquetDataBenchmark.readDecimalsIcebergVectorized5k      ss    5  10.508 ± 0.893   s/op
VectorizedReadFlatParquetDataBenchmark.readDecimalsSparkVectorized5k        ss    5   8.165 ± 1.205   s/op
VectorizedReadFlatParquetDataBenchmark.readDoublesIcebergVectorized5k       ss    5   2.880 ± 0.241   s/op
VectorizedReadFlatParquetDataBenchmark.readDoublesSparkVectorized5k         ss    5   2.610 ± 0.152   s/op
VectorizedReadFlatParquetDataBenchmark.readFloatsIcebergVectorized5k        ss    5   2.801 ± 0.376   s/op
VectorizedReadFlatParquetDataBenchmark.readFloatsSparkVectorized5k          ss    5   2.469 ± 0.155   s/op
VectorizedReadFlatParquetDataBenchmark.readIntegersIcebergVectorized5k      ss    5   2.286 ± 0.136   s/op
VectorizedReadFlatParquetDataBenchmark.readIntegersSparkVectorized5k        ss    5   2.580 ± 0.229   s/op
VectorizedReadFlatParquetDataBenchmark.readLongsIcebergVectorized5k         ss    5   2.684 ± 0.186   s/op
VectorizedReadFlatParquetDataBenchmark.readLongsSparkVectorized5k           ss    5   2.526 ± 0.767   s/op
VectorizedReadFlatParquetDataBenchmark.readStringsIcebergVectorized5k       ss    5   4.367 ± 0.207   s/op
VectorizedReadFlatParquetDataBenchmark.readStringsSparkVectorized5k         ss    5   3.964 ± 0.334   s/op
VectorizedReadFlatParquetDataBenchmark.readTimestampsIcebergVectorized5k    ss    5   1.434 ± 0.120   s/op
VectorizedReadFlatParquetDataBenchmark.readTimestampsSparkVectorized5k      ss    5   2.015 ± 0.351   s/op

after

Benchmark                                                                 Mode  Cnt   Score   Error  Units
VectorizedReadFlatParquetDataBenchmark.readDatesIcebergVectorized5k         ss    5   1.578 ± 0.045   s/op
VectorizedReadFlatParquetDataBenchmark.readDatesSparkVectorized5k           ss    5   1.757 ± 0.338   s/op
VectorizedReadFlatParquetDataBenchmark.readDecimalsIcebergVectorized5k      ss    5  10.965 ± 1.610   s/op
VectorizedReadFlatParquetDataBenchmark.readDecimalsSparkVectorized5k        ss    5   8.143 ± 0.300   s/op
VectorizedReadFlatParquetDataBenchmark.readDoublesIcebergVectorized5k       ss    5   2.791 ± 0.078   s/op
VectorizedReadFlatParquetDataBenchmark.readDoublesSparkVectorized5k         ss    5   2.623 ± 0.171   s/op
VectorizedReadFlatParquetDataBenchmark.readFloatsIcebergVectorized5k        ss    5   2.570 ± 0.065   s/op
VectorizedReadFlatParquetDataBenchmark.readFloatsSparkVectorized5k          ss    5   2.657 ± 0.692   s/op
VectorizedReadFlatParquetDataBenchmark.readIntegersIcebergVectorized5k      ss    5   2.651 ± 0.115   s/op
VectorizedReadFlatParquetDataBenchmark.readIntegersSparkVectorized5k        ss    5   2.547 ± 0.290   s/op
VectorizedReadFlatParquetDataBenchmark.readLongsIcebergVectorized5k         ss    5   2.305 ± 0.124   s/op
VectorizedReadFlatParquetDataBenchmark.readLongsSparkVectorized5k           ss    5   2.540 ± 0.076   s/op
VectorizedReadFlatParquetDataBenchmark.readStringsIcebergVectorized5k       ss    5   4.176 ± 0.222   s/op
VectorizedReadFlatParquetDataBenchmark.readStringsSparkVectorized5k         ss    5   4.231 ± 1.574   s/op
VectorizedReadFlatParquetDataBenchmark.readTimestampsIcebergVectorized5k    ss    5   1.597 ± 0.415   s/op
VectorizedReadFlatParquetDataBenchmark.readTimestampsSparkVectorized5k      ss    5   1.720 ± 0.052   s/op

I added decimal benchmark:
before:

Benchmark                                                                        Mode  Cnt   Score   Error  Units
VectorizedReadParquetDecimalBenchmark.readDecimalsIcebergVectorized5k              ss    5  10.623 ± 1.204   s/op
VectorizedReadParquetDecimalBenchmark.readDecimalsSparkVectorized5k                ss    5   8.322 ± 1.146   s/op
VectorizedReadParquetDecimalBenchmark.readIntBackedDecimalsIcebergVectorized5k     ss    5   1.593 ± 0.199   s/op
VectorizedReadParquetDecimalBenchmark.readIntBackedDecimalsSparkVectorized5k       ss    5   1.597 ± 0.064   s/op
VectorizedReadParquetDecimalBenchmark.readLongBackedDecimalsIcebergVectorized5k    ss    5   5.836 ± 0.406   s/op
VectorizedReadParquetDecimalBenchmark.readLongBackedDecimalsSparkVectorized5k      ss    5   3.285 ± 0.230   s/op

after

Benchmark                                                                        Mode  Cnt   Score   Error  Units
VectorizedReadParquetDecimalBenchmark.readDecimalsIcebergVectorized5k              ss    5  10.988 ± 2.355   s/op
VectorizedReadParquetDecimalBenchmark.readDecimalsSparkVectorized5k                ss    5   7.511 ± 0.484   s/op
VectorizedReadParquetDecimalBenchmark.readIntBackedDecimalsIcebergVectorized5k     ss    5   1.748 ± 0.174   s/op
VectorizedReadParquetDecimalBenchmark.readIntBackedDecimalsSparkVectorized5k       ss    5   1.631 ± 0.325   s/op
VectorizedReadParquetDecimalBenchmark.readLongBackedDecimalsIcebergVectorized5k    ss    5   3.623 ± 0.557   s/op
VectorizedReadParquetDecimalBenchmark.readLongBackedDecimalsSparkVectorized5k      ss    5   3.203 ± 0.209   s/op

rdblue · 2021-10-10T18:45:24Z

Running CI

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/GenericArrowVectorAccessorFactory.java

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java

spark/src/main/java/org/apache/iceberg/spark/source/BatchDataReader.java

rdblue · 2021-10-10T18:55:28Z

@ConeyLiu, this looks like an improvement, but I think that it relies too heavily on config properties that must be set the same way everywhere rather than detecting what should be done from inputs.

ConeyLiu · 2021-10-12T02:31:39Z

thanks @rdblue for the review. I will do some refactoring.

ConeyLiu · 2021-10-13T06:25:00Z

Hi @rdblue, please take a look again, thanks a lot. Also update benchmark results:

Benchmark                                                                 Mode  Cnt  Score   Error  Units
VectorizedReadFlatParquetDataBenchmark.readDatesIcebergVectorized5k         ss    5  1.320 ± 0.045   s/op
VectorizedReadFlatParquetDataBenchmark.readDatesSparkVectorized5k           ss    5  1.348 ± 0.044   s/op
VectorizedReadFlatParquetDataBenchmark.readDecimalsIcebergVectorized5k      ss    5  8.095 ± 0.308   s/op
VectorizedReadFlatParquetDataBenchmark.readDecimalsSparkVectorized5k        ss    5  8.013 ± 0.412   s/op
VectorizedReadFlatParquetDataBenchmark.readDoublesIcebergVectorized5k       ss    5  2.697 ± 0.113   s/op
VectorizedReadFlatParquetDataBenchmark.readDoublesSparkVectorized5k         ss    5  2.224 ± 0.071   s/op
VectorizedReadFlatParquetDataBenchmark.readFloatsIcebergVectorized5k        ss    5  2.135 ± 0.030   s/op
VectorizedReadFlatParquetDataBenchmark.readFloatsSparkVectorized5k          ss    5  2.233 ± 0.765   s/op
VectorizedReadFlatParquetDataBenchmark.readIntegersIcebergVectorized5k      ss    5  2.223 ± 0.043   s/op
VectorizedReadFlatParquetDataBenchmark.readIntegersSparkVectorized5k        ss    5  2.479 ± 0.096   s/op
VectorizedReadFlatParquetDataBenchmark.readLongsIcebergVectorized5k         ss    5  2.640 ± 0.041   s/op
VectorizedReadFlatParquetDataBenchmark.readLongsSparkVectorized5k           ss    5  2.498 ± 0.116   s/op
VectorizedReadFlatParquetDataBenchmark.readStringsIcebergVectorized5k       ss    5  4.043 ± 0.486   s/op
VectorizedReadFlatParquetDataBenchmark.readStringsSparkVectorized5k         ss    5  3.809 ± 0.163   s/op
VectorizedReadFlatParquetDataBenchmark.readTimestampsIcebergVectorized5k    ss    5  1.504 ± 0.298   s/op
VectorizedReadFlatParquetDataBenchmark.readTimestampsSparkVectorized5k      ss    5  1.856 ± 0.295   s/op

Benchmark                                                                        Mode  Cnt  Score   Error  Units
VectorizedReadParquetDecimalBenchmark.readDecimalsIcebergVectorized5k              ss    5  7.822 ± 0.441   s/op
VectorizedReadParquetDecimalBenchmark.readDecimalsSparkVectorized5k                ss    5  7.916 ± 0.908   s/op
VectorizedReadParquetDecimalBenchmark.readIntBackedDecimalsIcebergVectorized5k     ss    5  1.655 ± 0.130   s/op
VectorizedReadParquetDecimalBenchmark.readIntBackedDecimalsSparkVectorized5k       ss    5  1.539 ± 0.104   s/op
VectorizedReadParquetDecimalBenchmark.readLongBackedDecimalsIcebergVectorized5k    ss    5  3.468 ± 0.471   s/op
VectorizedReadParquetDecimalBenchmark.readLongBackedDecimalsSparkVectorized5k      ss    5  3.256 ± 0.587   s/op

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java

ConeyLiu · 2021-10-13T06:37:25Z

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java

            break;
          case UUID:
+          case FIXED_WIDTH_BINARY:
+          case FIXED_LENGTH_DECIMAL:


Here, combined the FIXED_WIDTH_BINARY to use fixedSizeBinaryBatchReader instead of fixedWidthTypeBinaryBatchReader. fixedWidthTypeBinaryBatchReader uses VarBinaryVector, it should use FixedSizeBinaryVector. Please correct me if I am wrong @nastra

yep those changes for FIXED_WIDTH_BINARY are correct. Note that support for FIXED is being added as part of #3029

However, I'm not sure we can change stuff for FIXED_LENGTH_DECIMAL. See also

iceberg/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedPageIterator.java

Lines 310 to 316 in 8058ec1

/**

* Method for reading a batch of decimals backed by fixed length byte array parquet data type. Arrow stores all

* decimals in 16 bytes. This method provides the necessary padding to the decimals read. Moreover, Arrow interprets

* the decimals in Arrow buffer as little endian. Parquet stores fixed length decimals as big endian. So, this method

* uses {@link DecimalVector#setBigEndian(int, byte[])} method so that the data in Arrow vector is indeed little

* endian.

*/

iceberg/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/VectorizedParquetDefinitionLevelReader.java

Line 362 in dbfa71e

valuesReader.getBuffer(typeWidth).get(byteArray, 0, typeWidth);

I think the valuesReader.getBuffer(typeWidth) shoud already return a little endian?

public ByteBuffer getBuffer(int length) { try { return valuesInputStream.slice(length).order(ByteOrder.LITTLE_ENDIAN); } catch (IOException e) { throw new ParquetDecodingException("Failed to read " + length + " bytes", e); } }

It seems like we don't need to DecimalVector#setBigEndian(int, byte[]).

And also, the UUID https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#uuid stored as big endian as well.

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/GenericArrowVectorAccessorFactory.java

nastra · 2021-10-14T09:49:05Z

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java

            break;
          case UUID:
+          case FIXED_WIDTH_BINARY:
+          case FIXED_LENGTH_DECIMAL:


yep those changes for FIXED_WIDTH_BINARY are correct. Note that support for FIXED is being added as part of #3029

nastra · 2021-10-14T09:59:56Z

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java

+  public VectorizedArrowReader(
+      ColumnDescriptor desc,
+      Types.NestedField icebergField,
+      Types.NestedField originalIcebergField,


@rdblue do we need to keep the original constructor without the originalIcebergField or could it be removed?

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java

ConeyLiu · 2021-11-04T07:25:19Z

@rdblue @nastra @pvary @kbendick Could you help to review when you have time? Thanks a lot.

rdblue · 2022-03-15T03:33:51Z

@ConeyLiu, could you rebase this? I'll take another look so we can get it in. Thanks!

site/docs/benchmarks.md

ConeyLiu · 2022-03-16T03:21:41Z

Thanks, @rdblue @kbendick, code has been rebased. Please take a look when you are free.

singhpk234

Thanks @ConeyLiu, for this change, I also stumbled on something similar recently.

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/GenericArrowVectorAccessorFactory.java

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java

jackye1995 · 2023-03-23T21:24:36Z

Sorry for the delayed review, was focusing on getting 1.2 release out. I think it's mostly there, added a few comments.

jackye1995 · 2023-03-23T21:39:26Z

@nastra could you also take another look?

ConeyLiu · 2023-03-27T07:23:57Z

Thanks, @jackye1995 @zhongyujiang for the review. Please take another look when you are free.

nastra

overall this LGTM, can we remove IntBackedDecimalBatchReader / LongBackedDecimalBatchReader / FixedLengthDecimalBatchReader / IntBackedDecimalPageReader / LongBackedDecimalPageReader?

@ConeyLiu could you please run the benchmark on your GH fork and post the link to it? I usually run benchmarks on my fork here and you should have the same action in your fork.

ConeyLiu · 2023-03-29T06:11:42Z

@nastra benchmark results at here

Benchmark                                                                  Mode  Cnt  Score   Error  Units
VectorizedReadFlatParquetDataBenchmark.readBigDecimalsIcebergVectorized5k    ss    5  9.512 ± 0.092   s/op
VectorizedReadFlatParquetDataBenchmark.readBigDecimalsSparkVectorized5k      ss    5  9.206 ± 0.145   s/op
VectorizedReadFlatParquetDataBenchmark.readDatesIcebergVectorized5k          ss    5  2.482 ± 0.187   s/op
VectorizedReadFlatParquetDataBenchmark.readDatesSparkVectorized5k            ss    5  2.492 ± 0.139   s/op
VectorizedReadFlatParquetDataBenchmark.readDecimalsIcebergVectorized5k       ss    5  5.154 ± 0.169   s/op
VectorizedReadFlatParquetDataBenchmark.readDecimalsSparkVectorized5k         ss    5  4.057 ± 0.079   s/op
VectorizedReadFlatParquetDataBenchmark.readDoublesIcebergVectorized5k        ss    5  4.546 ± 0.342   s/op
VectorizedReadFlatParquetDataBenchmark.readDoublesSparkVectorized5k          ss    5  3.897 ± 0.113   s/op
VectorizedReadFlatParquetDataBenchmark.readFloatsIcebergVectorized5k         ss    5  3.803 ± 0.096   s/op
VectorizedReadFlatParquetDataBenchmark.readFloatsSparkVectorized5k           ss    5  3.735 ± 0.185   s/op
VectorizedReadFlatParquetDataBenchmark.readIntegersIcebergVectorized5k       ss    5  4.021 ± 0.144   s/op
VectorizedReadFlatParquetDataBenchmark.readIntegersSparkVectorized5k         ss    5  3.493 ± 0.095   s/op
VectorizedReadFlatParquetDataBenchmark.readLongsIcebergVectorized5k          ss    5  4.056 ± 0.321   s/op
VectorizedReadFlatParquetDataBenchmark.readLongsSparkVectorized5k            ss    5  3.177 ± 0.214   s/op
VectorizedReadFlatParquetDataBenchmark.readStringsIcebergVectorized5k        ss    5  5.387 ± 0.089   s/op
VectorizedReadFlatParquetDataBenchmark.readStringsSparkVectorized5k          ss    5  6.683 ± 0.130   s/op
VectorizedReadFlatParquetDataBenchmark.readTimestampsIcebergVectorized5k     ss    5  2.283 ± 0.136   s/op
VectorizedReadFlatParquetDataBenchmark.readTimestampsSparkVectorized5k       ss    5  2.208 ± 0.091   s/op

Benchmark                                                                        Mode  Cnt   Score   Error  Units
VectorizedReadParquetDecimalBenchmark.readDecimalsIcebergVectorized5k              ss    5  12.878 ± 0.512   s/op
VectorizedReadParquetDecimalBenchmark.readDecimalsSparkVectorized5k                ss    5  11.764 ± 0.654   s/op
VectorizedReadParquetDecimalBenchmark.readIntBackedDecimalsIcebergVectorized5k     ss    5   2.711 ± 0.214   s/op
VectorizedReadParquetDecimalBenchmark.readIntBackedDecimalsSparkVectorized5k       ss    5   2.698 ± 0.165   s/op
VectorizedReadParquetDecimalBenchmark.readLongBackedDecimalsIcebergVectorized5k    ss    5   4.728 ± 0.310   s/op
VectorizedReadParquetDecimalBenchmark.readLongBackedDecimalsSparkVectorized5k      ss    5   5.356 ± 0.320   s/op

can we remove IntBackedDecimalBatchReader / LongBackedDecimalBatchReader / FixedLengthDecimalBatchReader / IntBackedDecimalPageReader / LongBackedDecimalPageReader?

Those methods/classes are public, is it safe to delete them? Or mark them as deprecated.

nastra · 2023-03-29T07:43:07Z

Those classes should be safe to delete. Also there are no API guarantees on iceberg-arrow, but just in case we could deprecate them and remove them in the next minor release, wdyt @jackye1995?

jackye1995 · 2023-03-29T14:43:03Z

Yes I agree, we can just mark it as deprecated and remove it after the next release just to follow the process

ConeyLiu · 2023-03-30T02:22:49Z

Thanks, @nastra @jackye1995 for your time. Those public methods/classes are marked as deprecated.

nastra · 2023-03-30T08:46:21Z

@ConeyLiu I think we should also deprecate IntBackedDecimalPageReader / LongBackedDecimalPageReader

ConeyLiu · 2023-03-30T11:02:55Z

@nastra updated, and also deprecate FixedLengthDecimalPageReader.

jackye1995

looks good to me, @nastra do you have any further comments?

jackye1995 · 2023-03-31T06:18:10Z

Thanks for the contribution! Thanks for the review @nastra !

ConeyLiu · 2023-03-31T08:19:10Z

Thanks all.

github-actions bot added arrow spark labels Oct 8, 2021

ConeyLiu force-pushed the vectorized branch from 826aa23 to 0ae42c5 Compare October 9, 2021 08:42

github-actions bot added the docs label Oct 9, 2021

rdblue reviewed Oct 10, 2021

View reviewed changes

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/GenericArrowVectorAccessorFactory.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 10, 2021

View reviewed changes

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/GenericArrowVectorAccessorFactory.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 10, 2021

View reviewed changes

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 10, 2021

View reviewed changes

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 10, 2021

View reviewed changes

spark/src/main/java/org/apache/iceberg/spark/source/BatchDataReader.java Outdated Show resolved Hide resolved

ConeyLiu commented Oct 13, 2021

View reviewed changes

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java Outdated Show resolved Hide resolved

ConeyLiu commented Oct 13, 2021

View reviewed changes

nastra reviewed Oct 14, 2021

View reviewed changes

ConeyLiu force-pushed the vectorized branch from 1ac3cfb to eac835a Compare November 4, 2021 07:23

ConeyLiu mentioned this pull request Mar 8, 2022

Slow performance on TPC-DS tests #4217

Closed

kbendick reviewed Mar 15, 2022

View reviewed changes

site/docs/benchmarks.md Outdated Show resolved Hide resolved

kbendick mentioned this pull request Mar 15, 2022

Spark - Update commands for running JMH tests with Spark 3.2 JMH #4328

Merged

ConeyLiu force-pushed the vectorized branch from eac835a to 9be401b Compare March 16, 2022 03:02

singhpk234 reviewed May 1, 2022

View reviewed changes

ConeyLiu force-pushed the vectorized branch from 82071fb to 3b1599b Compare May 5, 2022 12:55

ConeyLiu added 5 commits March 22, 2023 19:21

correct comments

8b4ba38

fixes

f2a0b5f

address comments

4fd8747

fixes code style

3ba01ff

update comments

f6a0b40

jackye1995 reviewed Mar 23, 2023

View reviewed changes

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/GenericArrowVectorAccessorFactory.java Outdated Show resolved Hide resolved

jackye1995 reviewed Mar 23, 2023

View reviewed changes

arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorHolder.java Show resolved Hide resolved

jackye1995 requested a review from nastra March 23, 2023 21:39

address comments

3ca3a3a

ConeyLiu force-pushed the vectorized branch from 4684365 to 3ca3a3a Compare March 27, 2023 06:58

nastra reviewed Mar 28, 2023

View reviewed changes

mark deprecated for unsed methods/classes

b6b9062

address comments

3fe2ae9

jackye1995 approved these changes Mar 31, 2023

View reviewed changes

nastra approved these changes Mar 31, 2023

View reviewed changes

jackye1995 merged commit 7c818c0 into apache:master Mar 31, 2023

ConeyLiu deleted the vectorized branch March 31, 2023 08:19

wypoon mentioned this pull request Oct 7, 2024

Arrow: Fix indexing in Parquet dictionary encoded values readers #11247

Merged

nastra mentioned this pull request Oct 11, 2024

Arrow: Deprecate unused fixed width binary reader classes #11292

Merged

	/**
	* Method for reading a batch of decimals backed by fixed length byte array parquet data type. Arrow stores all
	* decimals in 16 bytes. This method provides the necessary padding to the decimals read. Moreover, Arrow interprets
	* the decimals in Arrow buffer as little endian. Parquet stores fixed length decimals as big endian. So, this method
	* uses {@link DecimalVector#setBigEndian(int, byte[])} method so that the data in Arrow vector is indeed little
	* endian.
	*/

Optimized spark vectorized read parquet decimal #3249

Optimized spark vectorized read parquet decimal #3249

Uh oh!

Conversation

ConeyLiu commented Oct 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ConeyLiu commented Oct 8, 2021

Uh oh!

nastra commented Oct 8, 2021

Uh oh!

ConeyLiu commented Oct 9, 2021

Uh oh!

rdblue commented Oct 10, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdblue commented Oct 10, 2021

Uh oh!

ConeyLiu commented Oct 12, 2021

Uh oh!

ConeyLiu commented Oct 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ConeyLiu Oct 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nastra Oct 14, 2021

Choose a reason for hiding this comment

Uh oh!

nastra Oct 14, 2021

Choose a reason for hiding this comment

Uh oh!

ConeyLiu Oct 15, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nastra Oct 14, 2021

Choose a reason for hiding this comment

Uh oh!

nastra Oct 14, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ConeyLiu commented Nov 4, 2021

Uh oh!

rdblue commented Mar 15, 2022

Uh oh!

Uh oh!

ConeyLiu commented Mar 16, 2022

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jackye1995 commented Mar 23, 2023

Uh oh!

jackye1995 commented Mar 23, 2023

Uh oh!

ConeyLiu commented Mar 27, 2023

Uh oh!

nastra left a comment

Choose a reason for hiding this comment

Uh oh!

ConeyLiu commented Mar 29, 2023

Uh oh!

nastra commented Mar 29, 2023

Uh oh!

jackye1995 commented Mar 29, 2023

Uh oh!

ConeyLiu commented Oct 8, 2021 •

edited

Loading

ConeyLiu commented Oct 13, 2021 •

edited

Loading

ConeyLiu Oct 13, 2021 •

edited

Loading