Stripping AWS Mail component with TrueZIP blocks packaging for docker images

[poikilotherm]

Dear devs,
Dear @bsilverstein as the one who wrote the code for #3921,

while working on #4172, I am crafting a new Maven profile to generate smallest possible Docker images based on the payara 5 server-full image using https://dmp.fabric8.io.

As I want to make things really small, I don't want to bundle the dataverse dependencies within the WAR file, but create a small WAR without all deps, containing only classes and ressources. The deps get moved in by copying them the /lib of the Glassfish/Payara domain (so they are kind of "provided" scoped in Maven terminology). (For those interested in Docker: this should provide us with smaller layer changes and smaller hub transfers!)

The stripping of the mailing stuff from the AWS SDK makes this hard to achieve in two ways: a) I would to fiddle around with this, too and b) the TrueZIP plugin simply fails as I don't build the WAR with deps inside and it is not easy to skip it in a profile without making a mess in the pom.xml...

Back in #3921 where all of this was introduced, there has been a pointer to the AWS dev docs about using individual components.

Did you guys ever try to get that rolling? If not: maybe this would be the perfect time to do so now... Willing to contribute on this. 😄

---

This is related to a question I made on IRC (see logs).

EDIT:
@pdurbin suggested some clarification on IRC.

The pom.xml plugin section I would like to see removed:

-            <!-- v4.8: The truezip-maven-plugin below deletes two copies of a file that the AWS SDK bundle
-            includes called javamail.providers which breaks system emails. TODO: make our own AWS SDK without Amazon SES.
-            - bsilverstein 8/8/2017 -->
-            <plugin>
-                <groupId>org.codehaus.mojo</groupId>
-                <artifactId>truezip-maven-plugin</artifactId>
-                <version>1.1</version>
-                <executions>
-                    <execution>
-                        <id>remove-javamail-providers-from-exploded</id>
-                        <goals>
-                            <goal>remove</goal>
-                        </goals>
-                        <phase>package</phase>
-                        <configuration>
-                            <fileset>
-                                <directory>target/dataverse-${project.version}/WEB-INF/lib/aws-java-sdk-bundle-${aws.version}.jar/META-INF</directory>
-                                <includes>
-                                    <include>javamail.providers</include>
-                                </includes>
-                            </fileset>
-                        </configuration>
-                    </execution>
-                    <execution>
-                        <id>remove-javamail-providers-from-war</id>
-                        <goals>
-                            <goal>remove</goal>
-                        </goals>
-                        <phase>package</phase>
-                        <configuration>
-                            <fileset>
-                                <directory>target/dataverse-${project.version}.war/WEB-INF/lib/aws-java-sdk-bundle-${aws.version}.jar/META-INF</directory>
-                                <includes>
-                                    <include>javamail.providers</include>
-                                </includes>
-                            </fileset>
-                        </configuration>
-                    </execution>
-                </executions>
-            </plugin>

And something that needs to be changed (and complemented by some more stuff):

         <dependency>
             <groupId>com.amazonaws</groupId>
-            <artifactId>aws-java-sdk-bundle</artifactId>
+            <artifactId>aws-java-sdk-s3</artifactId>
             <version>${aws.version}</version>
         </dependency>

[matthew-a-dunlap]

Hey @poikilotherm , the reason we took this approach is due to dependency compatibility issues between the AWS SDK and Jackson/Glassfish. See around this post in 3921 for more info on that problem. The solution we went with was not the most elegant, but functioned well for our standard deployment strategy.

I am unaware at this point why we didn't go with pointing to the individual components, that seems like a good strategy to try. If you do try to fix this weird part of our build please make sure it still operates for our norm of keeping all the dependencies in the war. Thanks!

[poikilotherm]

Will have a look at the Jackson problems tomorrow... Thx for pointing that out.

Using the S3-only SDK seems to be alright, that's the only AWS stuff in use currently. This reduced the WAR size from 146MB to 128MB already, which seems good, right? 😉

I checked all of the involved jar archives, as far as I can tell, there is no javamail related stuff in there, thus the mail problems should be avoided thoroughly.

[poikilotherm]

Hi @matthew-a-dunlap,

you wrote the lines following

dataverse/pom.xml

Line 73 in c2dec15

<!-- Due to our old version of glassfish and its version of jackson,

. I'm not sure I fully understand what you meant with "an AWS SDK with unique references to needed libraries"... I guess this was related to the mentioned Jackson problems, right?

As far as I can see from the MVN AWS SDK, MVN AWS SDK Complete Bundle, MVN AWS SDK S3, there never have been explicit RUNTIME dependecies for the Jackson 2.6 stuff. EDIT: This is BS. Compile scope. My bad, sry. Anyway, there is no Jackson Databinding 2.6 on the classpath, see below.

There are compile scope deps against Jackson 2.6 in MVN AWS SDK Core. But since the release of .172, nowadays v.11.442, there have been only marginal changes in versioning and not for Jackson. So these "unique references" never were absent...?

What was the exact problem with Jackson you mentioned earlier?

The included Jackson Databinding in Glassfish 4.1 is v2.3.2 (lookup from the glassfish-4.1.zip JAR manifests). But this is overridden by jackson-databinding v2.5.3 within the WAR deps, which in turn is a dependency introduced by com.maxmind.geoip2. (Looked that up via mvn dependency:tree)

Included in the resulting dataverse WAR are the following Jackson JARs:

WEB-INF/lib/jackson-core-2.9.6.jar
WEB-INF/lib/jackson-dataformat-cbor-2.6.7.jar
WEB-INF/lib/jackson-annotations-2.9.6.jar
WEB-INF/lib/jackson-databind-2.5.3.jar
WEB-INF/lib/jackson-mapper-asl-1.9.7.jar
WEB-INF/lib/jackson-core-asl-1.9.7.jar

I'm puzzled...

[poikilotherm]

Maybe I have a hint about why you ran into problems and solved those when using the bundle: https://aws.amazon.com/de/blogs/developer/java-sdk-bundle/

To sum up that blog article: within the bundle, Jackson and other third party licenses are re-packaged under com.amazonaws.thirdparty and used inside. That would be a very good explanation why it started to "just work" when you switched to the complete bundle.

[poikilotherm]

Using the Maven Enforcer plugin and its dependency convergence rule leads to the following errors (among ALOT of others):

[WARNING] 
Dependency convergence error for com.fasterxml.jackson.core:jackson-databind:2.6.7.1 paths to dependency are:
+-edu.harvard.iq:dataverse:4.9.4
  +-com.amazonaws:aws-java-sdk-s3:1.11.172
    +-com.amazonaws:aws-java-sdk-core:1.11.172
      +-com.fasterxml.jackson.core:jackson-databind:2.6.7.1
and
+-edu.harvard.iq:dataverse:4.9.4
  +-com.amazonaws:aws-java-sdk-s3:1.11.172
    +-com.amazonaws:jmespath-java:1.11.172
      +-com.fasterxml.jackson.core:jackson-databind:2.6.7.1
and
+-edu.harvard.iq:dataverse:4.9.4
  +-com.maxmind.geoip2:geoip2:2.3.1
    +-com.maxmind.db:maxmind-db:1.0.0
      +-com.fasterxml.jackson.core:jackson-databind:2.4.2
and
+-edu.harvard.iq:dataverse:4.9.4
  +-com.maxmind.geoip2:geoip2:2.3.1
    +-com.fasterxml.jackson.core:jackson-databind:2.5.3
and
+-edu.harvard.iq:dataverse:4.9.4
  +-org.apache.tika:tika-parsers:1.19
    +-com.fasterxml.jackson.core:jackson-databind:2.9.6

And as we saw above, the v2.5.3 wins. This will take some work to sort out, especially if done for all defined deps (which is best practise anyway...).

[matthew-a-dunlap]

Hey @poikilotherm , sorry for the lack of info earlier, I had forgotten the details and some of them were lost to the sands of time and intern involvement. The movement of the 3rd party licenses to com.amazonaws.thirdparty was why we went with the big packaged library, as it was the most straightforward way of getting s3 operating. We talked about coming back to build our own specific copy of the s3 library but that's not been very high priority.

I'll bring it up during our team's tech hours today, there is definitely debt in cleaning up our dependencies in general and this one specifically.

[poikilotherm]

Most likely this is a matter of adding excludes to the dependency definitions or use dependencyManagement. See also http://timsteffens.blogspot.com/2014/05/solving-conflicts-with-transitive-maven.html?m=1 for managing these transitive deps.