Skip to content

Conversation

@JoshRosen
Copy link
Contributor

This pull request adds documentation on spark-redshift's transactional guarantees.

@codecov-io
Copy link

Current coverage is 88.99%

Merging #150 into master will not affect coverage as of 4990791

@@            master    #150   diff @@
======================================
  Files           13      13       
  Stmts          645     645       
  Branches       142     142       
  Methods          0       0       
======================================
  Hit            574     574       
  Partial          0       0       
  Missed          71      71       

Review entire Coverage Diff as of 4990791

Powered by Codecov. Updated on successful CI builds.

README.md Outdated
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I'm going to fix this in #151, I should probably just update this now to move the current text into a pre-1.6.1 caveat / callout.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we need to more specific on when eventual-consistency may have a impact because Amazon S3 buckets in all Regions provide read-after-write consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES. (https://aws.amazon.com/s3/faqs/)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that Amazon FAQ entry is really misleading / incomplete. My understanding of S3 read-after-write consistency is that

  1. it doesn't apply to bucket listing operations (see https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel), and
  2. while all regions provide read-after-write consistency, you have to use certain endpoints to receive those guarantees (specifically, I don't think that the global endpoint provides this for us-standard buckets (I'd clarify this with Amazon if you're really curious about the specifics)).

I don't know why the Amazon documentation doesn't just spell this out more clearly.

@JoshRosen
Copy link
Contributor Author

@yhuai, I've updated this to be organized slightly differently. PTAL.

@yhuai
Copy link

yhuai commented Jan 13, 2016

LGTM

@JoshRosen
Copy link
Contributor Author

Alright, I'm going to merge this now and will create issues to work on the future improvements promised in the text.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants