IAM Role credentials can expire between unload / load into Redshift

[cmkimerer]

Note: This is likely superseded by #199, but I'll file in case this fix needs to be carried over, or #199 doesn't land in the next release.

When using RedshiftWriter.saveToRedshift() the credentials are fetched early in the function, if the unloadData call takes a significant amount of time (

spark-redshift/src/main/scala/com/databricks/spark/redshift/RedshiftWriter.scala

Line 374 in cba5eee

val manifestUrl = unloadData(sqlContext, data, tempDir)

) it's possible that the credentials fetched have expired by the time doRedshiftLoad has run.

Ideally, the result returned from loadFromUri would return an instance of the InstanceProfileCredentialsProvider and getCredentials() would be called each time it was needed. This would handle the fact that the credentials can expire and would re-fetch when necessary. Obviously this change will break all other usages of fetching credentials right now, so perhaps an internal "credential wrapper" should be created to either return a wrapped BasicAWSCredentials when getCredentials() is called, or call through to the InstanceProfileCredentialsProvider.getCredentials() if that case is hit.

[JoshRosen]

Just merged a patch addressing #199, so I think that hopefully your use-case should be addressed in the next release.

I do agree that we should be smarter about how we handle instance profile credentials, though. The solution that we have today is quasi-functional but places a lot of burden on the user for managing credential refresh, etc; it would be great to have the library handle more of this itself.

[JoshRosen]

I've submitted #284 to fix this.