Skip to content

Conversation

@melin
Copy link

@melin melin commented Nov 13, 2020

update parquet version 1.11.1 and avro version 1.9.2,iceberg rely on these versions

update parquet version  1.11.1 and avro version 1.9.2
@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@github-actions github-actions bot added the BUILD label Nov 13, 2020
@dongjoon-hyun
Copy link
Member

cc @sunchao

@srowen
Copy link
Member

srowen commented Nov 15, 2020

I think this has more implications for compatibility with other libraries.
There are already JIRAs for this.
You need to update the deps/ files too.

@gemelen
Copy link
Contributor

gemelen commented Nov 18, 2020

@melin there is one more place to upgrade Avro dependency version, in project/plugins.sbt
Please, see https://github.com/apache/spark/blob/master/project/plugins.sbt#L37

@sunchao
Copy link
Member

sunchao commented Nov 18, 2020

Sorry for the late reply. Yes @melin : there's ongoing effort on this. Please see #27609 and #26804 for details.

To make the migration smooth we'll also need to upgrade Hive because 2.3.7 is using an old Avro version with incompatible API usage against Avro 1.9+. Therefore we're fixing that in Hive branch-2.3 and plan to do a new release after that. The details is in https://issues.apache.org/jira/browse/HIVE-21737.

@dongjoon-hyun
Copy link
Member

Thank you for making a PR, @melin . For now, I'll close this PR in favor of the prior approaches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants