[SPARK-20059][YARN] Use the correct classloader for HBaseCredentialProvider

[jerryshao]

What changes were proposed in this pull request?

Currently we use system classloader to find HBase jars, if it is specified by --jars, then it will be failed with ClassNotFound issue. So here changing to use child classloader.

Also putting added jars and main jar into classpath of submitted application in yarn cluster mode, otherwise HBase jars specified with --jars will never be honored in cluster mode, and fetching tokens in client side will always be failed.

How was this patch tested?

Unit test and local verification.

[SparkQA]

Test build #75052 has finished for PR 17388 at commit 9e15391.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

@vanzin @tgravescs @mridulm do you think it necessary to add additional jars and main jar into classloader for yarn cluster mode?

In my case I run Spark with HBase in secure cluster, so I need to specify hbase jars with --jars to make HBaseCredentailProvider work. But fortunately in yarn cluster mode, this jars are not added into classloader, so it will fail to get HBase token with class not found issue.

This also applies to the customized credential provider, if we write a customized one and package into main jar, then it will be failed to load by ServiceLoader because this main jar is not presented in client's classloader.

Though this could be fixed by expanding launch classpath (like SPARK_CLASSPATH) as a workaround, I think a good solution is to add to child's classpath.

What do you think, is there any concern to put these jars into child's classpath in yarn cluster mode? Thanks a lot.

[SparkQA]

Test build #75078 has finished for PR 17388 at commit ec48ccf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

I think that's ok; the jars are still added to a separate class loader, so they won't affect the Spark classes. The only caveat is that --jars also accepts remote paths, which won't work for the purpose of getting HBase tokens. But then those jars are also ignored for anything run in client mode.

[jerryshao]

Thanks @vanzin for your comments. Yes, remote jars will also not be added to client's classpath currently.

Any further comments?

[vanzin]

s/yarn#client/the YARN client/

Otherwise it seems you're talking about yarn-client mode.

[jerryshao]

Thanks @vanzin , just updated the comment.

[vanzin]

Minor comment on a comment, otherwise LGTM.

[SparkQA]

Test build #75334 has finished for PR 17388 at commit 92d9587.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Merging to master / 2.1.