Extend s3 credentials to use IAM instance profiles

[BaronWR]

At the moment, if you try to use spark-redshift on an ec2 instance without explicitly specifying AWS credentials, an error is thrown. It would be great if the S3Credentials class could try to get credentials from the instance profile, as with regular spark.

[JoshRosen]

Did you read the "AWS Credentials" section of the README? https://github.com/databricks/spark-redshift#aws-credentials

Can you elaborate more on how the approach outlined in the "IAM instance profiles" section is insufficient? I realize that we could automatically get the STS token so that the user doesn't have to specify it, so pull requests are welcome if you'd like to take that on.

[BaronWR]

Sorry, my mistake, somehow completely missed that. It would be nice if the process could be automated, either through STS (although I've run into issues trying to use the getSessionToken enpoint) or using something like InstanceProfileCredentialsProvider. I might take a look to see how complicated adding that would be.

[BaronWR]

Hmm, I've also run into #159 Is there a plan to release 0.6.1 soon? Thanks

[JoshRosen]

I'd like to try to get a new release out soon (it'll likely be 0.7.0, though, as I'm kinda-sorta trying to do a better approximation of semantic versioning) but things have been a bit delayed due to us being busy with other projects. In the meantime, though, you should be able to use SBT to build a development version and can use that in your application.

[JoshRosen]

By the way, automation of the STS credentials would be really helpful for Python users, for whom it's kind of tricky to call the Java SDKs for obtaining the tokens.

[avihoo]

Is there any expected release date to new version containing this STS fix?

[JoshRosen]

@avihoo, I just released v1.0.0 which should incorporate the STS fix: https://github.com/databricks/spark-redshift/releases/tag/v1.0.0

[avihoo]

Thank you very much!

[JoshRosen]

This should be done as of #277.