[Java] Upgrade Netty to 4.1.x

[asfimport]

The Netty community will declare Netty 4.0.x as EOL at the beginning of the second quarter of 2018: netty/netty#7439

It would be great that Arrow can migrate to Netty 4.1.x soon. This is the only blocker for Spark to migrate to Netty 4.1.x.

Reporter: Shixiong Zhu / @zsxwing
Assignee: Shixiong Zhu / @zsxwing

Related issues:

• Upgrade Arrow to 0.8.0 (blocks)
• Upgrade Netty version to 4.1.x final (blocks)

PRs and other links:

• GitHub Pull Request #1376

_{Note: This issue was originally created as ARROW-1864. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
@jacques-n how big of a project is this? Probably not something we could get into 0.8.0?

[asfimport]

Jacques Nadeau / @jacques-n:
Probably only a moderate amount of work. It's possible they made some things private that could cause problems but I'm not aware of anything offhand.

I probably can't get to it in the next week or two. If someone else wants to pick it up, switch the pom and the identify the issues I'd be to try to help give guidance on how to address any issues found.

[asfimport]

Shixiong Zhu / @zsxwing:
@wesm @jacques-n I made a PR to upgrade Netty. When will 0.8.0 be released? Is it possible to release it early December so that I have enough time to upgrade Arrow for Spark. I noticed there are a lot of APIs changes and upgrading Arrow from 0.4.0 to 0.8.0 for Spark will be non-trivial.

[asfimport]

Wes McKinney / @wesm:
Since there is quite a bit of work to do still for 0.8.0 I estimate that the release candidate will go out the week of December 11. I would suggest preparing a PR upgrading Spark's use of Arrow in the meantime rather than waiting for 0.8.0 to be released. I think that @BryanCutler and @icexelloss will be able to help

[asfimport]

Bryan Cutler / @BryanCutler:
Thanks for doing this @zsxwing! I have already started updating Spark for Arrow 0.8.0 and will submit a WIP PR under SPARK-22324 soon. I'll ping you when it's up if that will help test out the Netty upgrade.

[asfimport]

Shixiong Zhu / @zsxwing:
@BryanCutler Cool. You probably also need this PR which fixes all except Arrow for Netty 4.1.x: apache/spark#19829

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 1376
#1376