AWS: Change GlueCatalog skip archive default to true #6916
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
+1 for the changing default, also linking previous ticket #5965 (comment) should we update the doc as well in this PR. |
Contributor
Author
|
@singhpk234 Ah yes good point we should update the AWS integration docs in this PR, will update |
amogh-jahagirdar
force-pushed
the
glue-skip-archive-default
branch
from
221f64e to
ce863fc
Compare
amogh-jahagirdar
force-pushed
the
glue-skip-archive-default
branch
from
ce863fc to
ab5a7a1
Compare
jackye1995
approved these changes
Feb 23, 2023
Contributor
jackye1995
left a comment
jackye1995
left a comment
There was a problem hiding this comment.
thanks for raising this, this has been a complaint for a long time
singhpk234
approved these changes
Feb 23, 2023
Contributor
singhpk234
left a comment
singhpk234
left a comment
There was a problem hiding this comment.
LGTM, Thanks @amogh-jahagirdar !
yyanyy
approved these changes
Feb 23, 2023
Contributor
|
Thanks for the fix @amogh-jahagirdar , and thanks for @singhpk234 and @yyanyy for reviews! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently GLUE_CATALOG_SKIP_ARCHIVE_DEFAULT is set to false. This means every glue commit will archive older versions of table, which quickly hits the Glue limit (a max of 100k versions gets maintained) for streaming use cases. After discussing with users of Glue Catalog, came to the conclusion that it's better to change this default behavior.
CC: @jackye1995 @singhpk234