use reflection to call hadoop fs.rename to workaround different hadoop jar version in main and hdfs-storage extension class loader

[himanshug]

Fixes #3786 with updated version of #3787

Verified with hadoop-2.7.1 on a dev environment.

[b-slim]

👍 after tests.

[himanshug]

marking it WIP as it needs more testing and looks very hacky.

[himanshug]

@hamlet-lee in my testing, straight forward fs.getClass().getMethod(..) does not appear to work because rename method is not PUBLIC. This is noted in the javadocs too
. So, I had to add a recursive impl to go over all superclasses to find the rename method. See findRenameMethodRecursively(..) added in HadoopFsWrapper.java .

I wonder how it is working for you without this recursive search. Have you done anything else special in your setup?

[himanshug]

I tested current patch with our hadoop cluster, seems to be working fine.

[hamlet-lee]

@himanshug rename is PUBLIC in DistributedFileSystem.

public void rename(Path src, Path dst, final Rename... options) throws IOException {

[himanshug]

@hamlet-lee i see, it is increasing visibility in the override .. however, one of of my tests tested it for LocalFileSystem where it is not overridden to be made public. and in FileSystem it is declared to be protected.
So unfortunately, that assumption about it being public can't be made in general and recursion is still necessary.

thanks for pointing that out, now I understand why it worked for you without recursion.

[gianm]

@himanshug - I'm sad it's needed but thanks for writing it.