[MNG-6397] Maven Transitive Dependency Resolution Does Not Respect Repository Definition in pom.xml

[jira-importer]

Alan Czajkowski opened MNG-6397 and commented

Note: I am trying to do a build behind a firewall which means I cannot access the Internet, I can only access my internal Maven repository inside my network, so:

• grabbing artifacts from https://artifacts.example.com/repository/maven/ works fine
• grabbing artifacts from anywhere else fails due to firewall restrictions

Let's begin:

My pom.xml has the following:

...
    <dependencies>
...
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <version>2.0.0.RELEASE</version>
        </dependency>
...
    </dependencies>
...
    <repositories>
...
        <repository>
            <id>central</id>
            <name>Public</name>
            <url>https://artifacts.example.com/repository/maven/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
...
    </repositories>
...

The dependency:tree for the spring-boot-starter-web is as follows:

+- org.springframework.boot:spring-boot-starter-web:jar:2.0.0.RELEASE:compile
|  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.0.RELEASE:compile
|  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.4:compile
|  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.4:compile
|  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.4:compile
|  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.0.RELEASE:compile
|  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.28:compile
|  +- org.hibernate.validator:hibernate-validator:jar:6.0.7.Final:compile
|  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
|  |  +- org.jboss.logging:jboss-logging:jar:3.3.0.Final:compile
|  |  \- com.fasterxml:classmate:jar:1.3.1:compile
|  \- org.springframework:spring-web:jar:5.0.4.RELEASE:compile

How is it that the build fails as such:

...
Downloading: https://repo.spring.io/milestone/org/jboss/shrinkwrap/shrinkwrap-bom/1.2.3/shrinkwrap-bom-1.2.3.pom
Downloading: https://repo.spring.io/snapshot/org/jboss/shrinkwrap/shrinkwrap-bom/1.2.3/shrinkwrap-bom-1.2.3.pom
Downloading: https://dl.bintray.com/rabbitmq/maven-milestones/org/jboss/shrinkwrap/shrinkwrap-bom/1.2.3/shrinkwrap-bom-1.2.3.pom
Downloading: https://repo.maven.apache.org/maven2/org/jboss/shrinkwrap/shrinkwrap-bom/1.2.3/shrinkwrap-bom-1.2.3.pom
...
[ERROR] Failed to execute goal on project maven-multi-module-demo-backend: Could not resolve dependencies for project com.example.pipe:maven-multi-module-demo-backend:war:1.0.0-SNAPSHOT: Failed to collect dependencies at org.springframework.boot:spring-boot-starter-web:jar:2.0.0.RELEASE -> org.hibernate.validator:hibernate-validator:jar:6.0.7.Final: Failed to read artifact descriptor for org.hibernate.validator:hibernate-validator:jar:6.0.7.Final: Could not transfer artifact org.jboss.shrinkwrap:shrinkwrap-bom:pom:1.2.3 from/to spring-milestone (https://repo.spring.io/milestone): Connection reset -> [Help 1]
...

when I did not even reference this repo spring-milestone ([https://repo.spring.io/milestone]) anywhere in my pom.xml?

When you go down the Spring Boot rabbit hole (go into the spring-boot-starter-web's pom.xml and then traverse up its parent-pom structure a few jumps) you'll eventually get to a parent-pom spring-boot-dependencies with this definition:

...
    <repositories>
        <repository>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
            <id>spring-milestone</id>
            <name>Spring Milestone</name>
            <url>https://repo.spring.io/milestone</url>
        </repository>
        <repository>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
            <id>spring-snapshot</id>
            <name>Spring Snapshot</name>
            <url>https://repo.spring.io/snapshot</url>
        </repository>
        <repository>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
            <id>rabbit-milestone</id>
            <name>Rabbit Milestone</name>
            <url>https://dl.bintray.com/rabbitmq/maven-milestones</url>
        </repository>
    </repositories>
...

How is it that the Maven build does not even attempt to reach out to https://artifacts.example.com/repository/maven/ to try to find the missing dependency shrinkwrap-bom? and only reaches out to the above repos only and not the one defined in my own pom.xml?

This seems like a transitive dependency resolution bug to me as the Maven build does not even make a single attempt at trying to get shrinkwrap-bom from the <repository> that I have explicitly defined in my pom.xml. The (grand)parents of the spring-boot-starter-web dependency completely hi-jack the repository list that the build pulls from (this type of hi-jacking should not be allowed). The shrinkwrap-bom artifact does exist in [https://artifacts.example.com/repository/maven/] and can be fetched no problem if it is explicitly defined in my pom.xml but defining it explicitly would be a work-around and I cannot use this work-around in my situation.

---

Affects: 3.0, 3.5.0, 3.5.2, 3.5.3, 3.6.0, 3.6.1, 3.6.3

Issue Links:

• MNG-6772 Super POM overwrites remapped central repository in nested dependencyManagement import POMs
  ("is caused by")

• MRESOLVER-168 add DEBUG message when downloading an artifact from repositories

• MNG-7094 Assure in an IT that reactor repository definitions are not used when Model Builder downloads dependencies with scope import

Backported to: waiting-for-feedback

1 votes, 15 watchers

[jira-importer]

Karl Heinz Marbaise commented

You should configure your repository manager correctly in your settings.xml and define a
mirrorOf entry to redirect all request to your internal repository manager ...

It also interesting to see things like this: : Connection reset -> [Help 1]... ?

[jira-importer]

Alan Czajkowski commented

Karl Heinz Marbaise

1. I appreciate your "work around" suggestion, and I've already thought of trying this, but this is not an acceptable solution for me as this build goes into a build system where I am not able to modify any settings.xml file. Everything related to the build must be completely self-contained inside the pom.xml, as this is a natural expectation of Maven, hence why I am logging this as a "bug" because the build should work naturally without me having to define mirrors.

2. It is not interesting to see Connection reset because at the very beginning of the description I explained the firewall situation. The firewall here is not causing anything to break in Maven. Maven is simply ignoring/not respecting my <repository> definition in my own pom.xml.

[jira-importer]

Alan Czajkowski commented

Work-around #1:

• use mirrors inside settings.xml

...
<mirrors>
  <mirror>
    <id>mirror-1</id>
    <name>Mirror - 1</name>
    <url>https://artifacts.example.com/repository/maven/</url>
    <mirrorOf>spring-milestone</mirrorOf>
  </mirror>
  <mirror>
    <id>mirror-2</id>
    <name>Mirror - 2</name>
    <url>https://artifacts.example.com/repository/maven/</url>
    <mirrorOf>spring-snapshot</mirrorOf>
  </mirror>
  <mirror>
    <id>mirror-3</id>
    <name>Mirror - 3</name>
    <url>https://artifacts.example.com/repository/maven/</url>
    <mirrorOf>rabbit-milestone</mirrorOf>
  </mirror>
</mirrors>
...

Work-around #2:

• restate the hi-jacking repositories inside your pom.xml, keep the <id> tags the same, and then do your own hi-jacking of the <url> tags inside the <repository> definitions, like so:

...
<repository>
  <snapshots>
    <enabled>false</enabled>
  </snapshots>
  <id>spring-milestone</id>
  <name>Spring Milestone</name>
  <!-- # hi-jacked URL -->
  <url>https://artifacts.example.com/repository/maven/</url>
</repository>
<repository>
  <snapshots>
    <enabled>true</enabled>
  </snapshots>
  <id>spring-snapshot</id>
  <name>Spring Snapshot</name>
  <!-- # hi-jacked URL -->
  <url>https://artifacts.example.com/repository/maven/</url>
</repository>
<repository>
  <snapshots>
    <enabled>false</enabled>
  </snapshots>
  <id>rabbit-milestone</id>
  <name>Rabbit Milestone</name>
  <!-- # hi-jacked URL -->
  <url>https://artifacts.example.com/repository/maven/</url>
</repository>
...

[jira-importer]

Alan Czajkowski commented

is anybody looking into this bug?

[jira-importer]

Karl Heinz Marbaise commented

If you don't like my "work around" sorry...the settings.xml is the solution to that particular problem. If you have already a build system this should already being configured correctly that way otherwise it not correctly configured (for example you should use config file provider plugin which offers this exactly for the settings.xml) ...

I would suggest to define a <mirrorOf>*</mirrorOf> which is easier...

The point is that you are expecting to overwrite the repository definitions of the dependencies/transitive dependencies which is not possible by using a repository entry in the pom.

The point is that each pom/artifact maintainer can put repository definitions into their pom file (which is not a good idea cause it exactly happens what here is described...different story)...which results in exactly this situation. There for you have the option via settings.xml and define a mirror which redirects all request from each pom through the given mirror.

If you don't like the solution or is not acceptable for you than please provide a patch(es) to solve it the way you like it...

[jira-importer]

Alan Czajkowski commented

Karl Heinz Marbaise
To be clear: this is a bug that needs to be fixed

"The point is that you are expecting to overwrite the repository definitions of the dependencies/transitive dependencies which is not possible by using a repository entry in the pom." <-- this is not true, my simple expectation is that the Maven build at least try to attempt to reach out to the repo that i have in my own pom.xml alongside the repos defined in the dependency pom.xml <-- this is a reasonable expectation of Maven

Your dependencies whether transitive or not, should not be allowed to do a complete hi-jacking of the list of repositories that the Maven build pulls from, which is what is happening here <-- that is the bug, the hi-jacking part

the mirror work-around is not possible as i am not allowed to modify the current build system in place, which means the only way forward is to either explicitly define all of the problematic dependencies in my pom.xml (a very tedious and horrible solution), or do the second work-around i mentioned above by re-hi-jacking the repos again ... none of these work-arounds are ideal, nor do they follow reasonable expectations of Maven

[jira-importer]

Alan Czajkowski commented

this is a common problem that comes up regularly in the various projects that I work on (especially projects that use Spring Boot and the Spring Boot parent in their projects), and I have to use the mirror work-around in the settings.xml (which is a poor solution)

is anybody looking into this?

[jira-importer]

Eddie Wiegers commented

I just ran into a similar-sounding issue that based on your build output looks like it may be the same root cause. I have logged a separate issue (MNG-6772) to provide my investigation notes, but wanted to give a heads up that it looks like the same root issue.

[jira-importer]

Alan Czajkowski commented

Eddie Wiegers unfortunately, I don't think anybody is looking into this or taking it seriously :(

[jira-importer]

Eddie Wiegers commented

I submitted a fix, so if it gets accepted hopefully this will be resolved in Maven 3.6.3.

[jira-importer]

Alan Czajkowski commented

Eddie Wiegers amazing, thank you!

[jira-importer]

Fan Guogang commented

mirrorOf * is not a good option, for example if you use a hosted repository server, such as nexus or artifactory, you need to create a group contains all repositories,that will cause slow performance when resloving dependencies, because nexus will need to contact all repositories to get metadata.

[jira-importer]

Alan Czajkowski commented

Karl Heinz Marbaise can we get Eddie Wiegers's fix accepted?

[jira-importer]

Elliotte Rusty Harold commented

sounds like the decision here is won't fix, WAI. Should we close the bug?

[jira-importer]

Alan Czajkowski commented

why would this be closed? this is a legitimate problem with Maven dependency resolution, and becoming a more prominent problem with the proliferation of Spring Boot