Add ability to handle s3a URIs

[drcrallen]

More recent implementations of Hadoop's FS use s3a scheme for native Amazon library access to S3.

There are a few places in the druid code that assume a file is either s3 or s3n to use the s3 extensions.

This ask is that the uri scheme for s3a is properly handled.

[saurabhere]

Is there any ETA/ release version for this feature to be implemented yet?
We also need to access S3 for hadoop ingestion and always passing AWS credentials is not safe. Can it at least be hidden in the task logs?

[gianm]

@saurabhere no ETA that I am aware of.

You should be able to work around this, though, by setting fs.s3.impl and fs.s3n.impl both to "org.apache.hadoop.fs.s3a.S3AFileSystem" and possibly setting some other properties too. That should get the system into using s3a for all s3:// and s3n:// urls, and it should be able to use your instance roles.

[saurabhere]

I tried that.
I got

Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) ~[?:?]

So, I downloaded hadoop-aws-2.7.3 explicitely into hadoop-dependencies
Then, i got

Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:285) ~[?:?]
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) ~[?:?]

This takes me to https://issues.apache.org/jira/browse/HADOOP-12420
Then I tried copying aws-java-sdk-1.7.4 to the hadoop-dependencies in vain.
Is there something else I can do?

[saurabhere]

So, I built my own version of hadoop-aws-2.7.3 by changing the aws-java-sdk version dependency to 1.10.21 and updating the call to TransferManagerConfiguration.setMultipartUploadThreshold(L),
and it works smoothly.
Thanks @gianm for the vote of confidence. 😁

[gianm]

Yay 😄

I'd like to add official support for this at some point but I'm glad you found a workaround.

[robertervin]

+1

[b-slim]

related to #3940

[dene14]

@drcrallen shouldn't it be closed now?

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.