[SPARK-27733][CORE] Upgrade Avro to version 1.10.1 #31232

iemejia · 2021-01-18T09:40:28Z

What changes were proposed in this pull request?

Update Avro dependency to version 1.10.1

Why are the changes needed?

To catch up multiple improvements of Avro as well as fix security issues on transitive dependencies.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Since there were no API changes required we just run the tests

.github/PULL_REQUEST_TEMPLATE

dev/deps/spark-deps-hadoop-2.7-hive-2.3

wangyum · 2021-01-18T09:46:55Z

core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java

Why do we need this change?

This is an improvement because Nullable in that class is used to designate class attributes that are nullable (like javax.annotation) and not to refer to nullability of an Avro field.

wangyum · 2021-01-18T10:05:21Z

ok to test.

SparkQA · 2021-01-18T11:45:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38778/

SparkQA · 2021-01-18T12:19:45Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38778/

HyukjinKwon · 2021-01-18T12:25:13Z

@iemejia, please fill the PR description.

SparkQA · 2021-01-18T13:47:12Z

Test build #134193 has finished for PR 31232 at commit cb3b0ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

iemejia · 2021-01-18T14:00:01Z

@iemejia, please fill the PR description.

Done thanks for the reminder @HyukjinKwon

iemejia · 2021-01-18T14:00:42Z

All green now PTAL
R: @wangyum @dongjoon-hyun

dongjoon-hyun

Thank you for pinging me, @iemejia .

cc @gengliangwang

dev/deps/spark-deps-hadoop-2.7-hive-2.3

gengliangwang · 2021-01-19T05:49:12Z

@iemejia Thanks for the work!
I am +1 with the upgrade. The bug fix from apache/avro#969 is also important.
Please address comments and keep Avro upgrade related changes only in this PR.

SparkQA · 2021-01-19T10:44:56Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38806/

SparkQA · 2021-01-19T11:14:30Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38806/

SparkQA · 2021-01-19T12:44:47Z

Test build #134221 has finished for PR 31232 at commit 6c6b137.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

iemejia · 2021-01-19T14:43:26Z

Hive error looks like a flake is there a way to re run that CI job/action?

mridulm · 2021-01-19T16:27:14Z

+CC @xkrogen

SparkQA · 2021-01-20T09:49:08Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38846/

SparkQA · 2021-01-20T09:57:56Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38846/

iemejia · 2021-01-20T10:02:28Z

Yay! green again after rebasing. PTAL again

SparkQA · 2021-01-20T11:38:43Z

Test build #134260 has finished for PR 31232 at commit f0ea522.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang

LGTM. Again, thanks for the work!

gengliangwang · 2021-01-20T11:46:04Z

Also cc @bozhang2820

wangyum · 2021-01-20T15:58:07Z

AvroReadBenchmark result.
Avro 1.8.2:

================================================================================================
SQL Single Numeric Column Scan
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single TINYINT Column Scan:           Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2478           2526          68          6.3         157.5       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single SMALLINT Column Scan:          Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2481           2523          59          6.3         157.8       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single INT Column Scan:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2416           2446          42          6.5         153.6       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single BIGINT Column Scan:            Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2757           2794          53          5.7         175.3       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single FLOAT Column Scan:             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2540           2570          41          6.2         161.5       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single DOUBLE Column Scan:            Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2570           2583          19          6.1         163.4       1.0X


================================================================================================
Int and String Scan
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Int and String Scan:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of columns                                     4050           4093          62          2.6         386.2       1.0X


================================================================================================
Partitioned Table Scan
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Partitioned Table:                        Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Data column                                        2880           2897          24          5.5         183.1       1.0X
Partition column                                   2522           2532          14          6.2         160.3       1.1X
Both columns                                       2936           2945          14          5.4         186.6       1.0X


================================================================================================
Repeated String Scan
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Repeated String:                          Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length                               2923           3065         201          3.6         278.7       1.0X


================================================================================================
String with Nulls Scan
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
String with Nulls Scan (0.0%):            Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length                               4868           4944         108          2.2         464.3       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
String with Nulls Scan (50.0%):           Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length                               3302           3317          22          3.2         314.9       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
String with Nulls Scan (95.0%):           Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length                               2052           2080          39          5.1         195.7       1.0X


================================================================================================
Single Column Scan From Wide Columns
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Single Column Scan from 100 columns:      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of single column                               3851           3858          10          0.3        3672.5       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Single Column Scan from 200 columns:      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of single column                               7399           7505         149          0.1        7056.6       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Single Column Scan from 300 columns:      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of single column                              11134          11168          48          0.1       10618.4       1.0X


Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Filters pushdown:                         Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
w/o filters                                        8049           8189         145          0.1        8049.3       1.0X
pushdown disabled                                  7728           7784          57          0.1        7728.5       1.0X
w/ filters                                         3291           3310          27          0.3        3291.3       2.4X

Avro 1.10.1:

================================================================================================
SQL Single Numeric Column Scan
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single TINYINT Column Scan:           Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2513           2550          53          6.3         159.8       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single SMALLINT Column Scan:          Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2451           2505          77          6.4         155.8       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single INT Column Scan:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2474           2476           3          6.4         157.3       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single BIGINT Column Scan:            Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2845           2860          21          5.5         180.9       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single FLOAT Column Scan:             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2507           2513           8          6.3         159.4       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
SQL Single DOUBLE Column Scan:            Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum                                                2527           2531           6          6.2         160.7       1.0X


================================================================================================
Int and String Scan
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Int and String Scan:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of columns                                     4172           4179           9          2.5         397.9       1.0X


================================================================================================
Partitioned Table Scan
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Partitioned Table:                        Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Data column                                        2848           2878          43          5.5         181.1       1.0X
Partition column                                   2526           2557          43          6.2         160.6       1.1X
Both columns                                       2967           2967           1          5.3         188.6       1.0X


================================================================================================
Repeated String Scan
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Repeated String:                          Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length                               2788           2868         113          3.8         265.9       1.0X


================================================================================================
String with Nulls Scan
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
String with Nulls Scan (0.0%):            Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length                               4594           4667         103          2.3         438.1       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
String with Nulls Scan (50.0%):           Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length                               2993           2998           8          3.5         285.4       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
String with Nulls Scan (95.0%):           Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of string length                               1664           1675          16          6.3         158.7       1.0X


================================================================================================
Single Column Scan From Wide Columns
================================================================================================

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Single Column Scan from 100 columns:      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of single column                               3845           3860          22          0.3        3666.6       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Single Column Scan from 200 columns:      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of single column                               7552           7562          15          0.1        7202.0       1.0X

Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Single Column Scan from 300 columns:      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Sum of single column                              11295          11337          60          0.1       10771.6       1.0X


Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11 on Linux 3.10.0-957.10.1.el7.x86_64
Intel Core Processor (Broadwell, IBRS)
Filters pushdown:                         Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
w/o filters                                        8222           8378         171          0.1        8222.4       1.0X
pushdown disabled                                  8089           8307         293          0.1        8089.4       1.0X
w/ filters                                         3402           3470          67          0.3        3402.3       2.4X

dongjoon-hyun

+1, LGTM. Thank you, @iemejia and all!
Merged to master for Apache Spark 3.2.0.

dongjoon-hyun · 2021-01-20T23:44:41Z

@iemejia I added you to the Apache Spark contributor group and assigned SPARK-27733 to you. Thanks!

iemejia · 2021-01-21T08:10:49Z

Thanks to you @dongjoon-hyun and @wangyum! This PR seems 'trivial' but when you have lived through the multi project issues/fixes and dependencies you realize this would not have been possible without everybody's work.

iemejia · 2021-01-21T08:34:01Z

@wangyum just for learning's sake what command do you use to calculate the benchmarks?

wangyum · 2021-01-21T08:46:32Z

@iemejia

spark/external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroReadBenchmark.scala

Lines 29 to 41 in d897825

    
           /** 
        
            * Benchmark to measure Avro read performance. 
        
            * {{{ 
        
            *   To run this benchmark: 
        
            *   1. without sbt: bin/spark-submit --class <this class> 
        
            *        --jars <catalyst test jar>,<core test jar>,<spark-avro jar> <avro test jar> 
        
            *   2. build/sbt "avro/test:runMain <this class>" 
        
            *   3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "avro/test:runMain <this class>" 
        
            *      Results will be written to "benchmarks/AvroReadBenchmark-results.txt". 
        
            * }}} 
        
            */ 
        
           object AvroReadBenchmark extends SqlBasedBenchmark {

iemejia · 2021-01-21T21:07:11Z

Thanks for the ref 👍 @wangyum !

### What changes were proposed in this pull request? Update Avro dependency to version 1.10.1 ### Why are the changes needed? To catch up multiple improvements of Avro as well as fix security issues on transitive dependencies. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Since there were no API changes required we just run the tests Closes apache#31232 from iemejia/SPARK-27733-avro-upgrade. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

Update Avro dependency to version 1.10.1 To catch up multiple improvements of Avro as well as fix security issues on transitive dependencies. No Since there were no API changes required we just run the tests Closes apache#31232 from iemejia/SPARK-27733-avro-upgrade. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

iemejia mentioned this pull request Jan 18, 2021

[WIP][SPARK-27733][CORE] Upgrade Avro to 1.9.2 #27609

Closed

wangyum reviewed Jan 18, 2021

View reviewed changes

.github/PULL_REQUEST_TEMPLATE Outdated Show resolved Hide resolved

wangyum reviewed Jan 18, 2021

View reviewed changes

dev/deps/spark-deps-hadoop-2.7-hive-2.3 Outdated Show resolved Hide resolved

wangyum reviewed Jan 18, 2021

View reviewed changes

github-actions bot added AVRO BUILD CORE DOCS INFRA SQL labels Jan 18, 2021

dongjoon-hyun reviewed Jan 18, 2021

View reviewed changes

wangyum reviewed Jan 19, 2021

View reviewed changes

dev/deps/spark-deps-hadoop-2.7-hive-2.3 Outdated Show resolved Hide resolved

wangyum reviewed Jan 19, 2021

View reviewed changes

dev/deps/spark-deps-hadoop-2.7-hive-2.3 Outdated Show resolved Hide resolved

iemejia force-pushed the SPARK-27733-avro-upgrade branch from cb3b0ce to 6c6b137 Compare January 19, 2021 09:47

iemejia mentioned this pull request Jan 19, 2021

Schema registry should not include classes in the org.apache.avro package confluentinc/schema-registry#1432

Closed

[SPARK-27733][CORE] Upgrade Avro to version 1.10.1

f0ea522

iemejia force-pushed the SPARK-27733-avro-upgrade branch from 6c6b137 to f0ea522 Compare January 20, 2021 08:31

gengliangwang approved these changes Jan 20, 2021

View reviewed changes

heuermh mentioned this pull request Jan 20, 2021

[SPARK-26346][BUILD][SQL] Upgrade Parquet to 1.11.1 #26804

Closed

wangyum approved these changes Jan 20, 2021

View reviewed changes

dongjoon-hyun approved these changes Jan 20, 2021

View reviewed changes

dongjoon-hyun closed this in e9e81f7 Jan 20, 2021

iemejia deleted the SPARK-27733-avro-upgrade branch January 21, 2021 08:04

LorenzoMartini mentioned this pull request Apr 19, 2021

[SPARK-27733][CORE] Upgrade Avro to version 1.10.1 palantir/spark#757

Merged

[SPARK-27733][CORE] Upgrade Avro to version 1.10.1 #31232

[SPARK-27733][CORE] Upgrade Avro to version 1.10.1 #31232

Uh oh!

Conversation

iemejia commented Jan 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

wangyum Jan 18, 2021

Choose a reason for hiding this comment

Uh oh!

iemejia Jan 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangyum commented Jan 18, 2021

Uh oh!

SparkQA commented Jan 18, 2021

Uh oh!

SparkQA commented Jan 18, 2021

Uh oh!

HyukjinKwon commented Jan 18, 2021

Uh oh!

SparkQA commented Jan 18, 2021

Uh oh!

iemejia commented Jan 18, 2021

Uh oh!

iemejia commented Jan 18, 2021

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gengliangwang commented Jan 19, 2021

Uh oh!

SparkQA commented Jan 19, 2021

Uh oh!

SparkQA commented Jan 19, 2021

Uh oh!

SparkQA commented Jan 19, 2021

Uh oh!

iemejia commented Jan 19, 2021

Uh oh!

mridulm commented Jan 19, 2021

Uh oh!

SparkQA commented Jan 20, 2021

Uh oh!

SparkQA commented Jan 20, 2021

Uh oh!

iemejia commented Jan 20, 2021

Uh oh!

SparkQA commented Jan 20, 2021

Uh oh!

gengliangwang left a comment

Choose a reason for hiding this comment

Uh oh!

gengliangwang commented Jan 20, 2021

Uh oh!

wangyum commented Jan 20, 2021

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jan 20, 2021

Uh oh!

iemejia commented Jan 21, 2021

Uh oh!

iemejia commented Jan 21, 2021

Uh oh!

wangyum commented Jan 21, 2021

Uh oh!

iemejia commented Jan 21, 2021

Uh oh!

Reviewers

iemejia commented Jan 18, 2021 •

edited

Loading

iemejia Jan 18, 2021 •

edited

Loading