-
Notifications
You must be signed in to change notification settings - Fork 347
Add documentation on transactional guarantees #150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Current coverage is
|
README.md
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since I'm going to fix this in #151, I should probably just update this now to move the current text into a pre-1.6.1 caveat / callout.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we need to more specific on when eventual-consistency may have a impact because Amazon S3 buckets in all Regions provide read-after-write consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES. (https://aws.amazon.com/s3/faqs/)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that Amazon FAQ entry is really misleading / incomplete. My understanding of S3 read-after-write consistency is that
- it doesn't apply to bucket listing operations (see https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel), and
- while all regions provide read-after-write consistency, you have to use certain endpoints to receive those guarantees (specifically, I don't think that the global endpoint provides this for us-standard buckets (I'd clarify this with Amazon if you're really curious about the specifics)).
I don't know why the Amazon documentation doesn't just spell this out more clearly.
|
@yhuai, I've updated this to be organized slightly differently. PTAL. |
|
LGTM |
|
Alright, I'm going to merge this now and will create issues to work on the future improvements promised in the text. |
This pull request adds documentation on
spark-redshift's transactional guarantees.