fix distribution to not include contrib extensions by default, don't …#6494
Merged
gianm merged 1 commit intoapache:masterfrom Oct 19, 2018
dclim:distribution-fixes
Merged
fix distribution to not include contrib extensions by default, don't …#6494gianm merged 1 commit intoapache:masterfrom dclim:distribution-fixes
gianm merged 1 commit intoapache:masterfrom
dclim:distribution-fixes
Conversation
…pull the entire AWS SDK bundle
Contributor
|
@dclim thanks for doing this! I think removing the Dependency on Mysql is the way to go. Hive does that for Mysql, in fact we do not use it that much in the code. Same with LZO libs for hadoop/spark. |
dclim
added a commit
that referenced
this pull request
Oct 21, 2018
…pull the entire AWS SDK bundle (#6494)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…pull the entire AWS SDK bundle
When trying to push the 0.13 RC1 binary to the Apache repository, I was getting rejected, likely due to the file being too large (it was 385MB). While investigating the bloat, I found two major factors:
I fixed the first one by splitting off a separate
bundle-contrib-extsprofile as previous which can be used to add the contrib extensions; otherwise only the core extensions are bundled.For the second one, I removed the dependency on
aws-java-sdk-bundleand only pulledaws-java-sdk-ec2andaws-java-sdk-s3. I am not sure if the full bundle was pulled in for any particular reason or just as a convenience.I also excluded a few larger dependencies from extensions that were already included in the core.
Binary tarball reduced in size from 385MB to 210MB.
FYI, I am still waiting on a response to a JIRA ticket to Apache Legal regarding how the MySQL connector should be packaged: https://issues.apache.org/jira/browse/LEGAL-423. If I don't hear back soon, I'll try to push RC1 (for the third time!) and do an RC2 if they express concerns with our current packaging system. My hunch is that we will be required to separate the offending library (GPL-licensed Java MySQL connector library) from the rest of the MySQL extension.. so if others feel good about it, I wouldn't be opposed to going ahead and making the change.