[MRESOLVER-7] Download dependency POMs in parallel in BF collector

[jira-importer]

Harald Wellmann opened MRESOLVER-7 and commented

Background

When building a project with dependencies not yet available in the local repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the dependency POMs sequentially and then proceeds downloading the dependency JARs with up to 5 threads in parallel.

Due to this, when first building a project with a large number of dependencies, downloading a large number of small POMs may take a lot longer than downloading the much larger JARs, or even longer than building the project itself, especially when a repository manager is used which increases the download latency.

Enhancement

Download POMs of (transitive) dependencies in parallel to significantly speed up initial builds of large projects.

---

Affects: Aether 1.0.2

Attachments:

• resolve_deps.png (38.74 kB)
• resolver.log (14.49 kB)

Issue Links:

• MNG-5896 Download dependency POMs in parallel
  ("blocks")

• MNG-6958 mvn only uses a single thread for downloading poms.
  ("is duplicated by")

• MNG-6672 Upgrade Maven Resolver to 1.4.2
  ("is part of")

• MRESOLVER-92 Revert MRESOLVER-7
  ("causes")

• MNG-6712 Downgrade maven-resolver:1.4.0 to 1.3.3

• MRESOLVER-32 Support parallel artifact/metadata uploads

Remote Links:

• GitHub Pull Request #10
  
• GitHub Pull Request #142
  
• GitHub Pull Request #178
  
• GitHub Pull Request #2
  
• GitHub Pull Request #30
  

24 votes, 34 watchers

[jira-importer]

Harald Wellmann commented

Pull request:

#2

[jira-importer]

Michael Osipov commented

Please raise the review on the dev mailing list, we'd like to push a release this year.

[jira-importer]

Sylwester Lachiewicz commented

rebased Herald's version splitted to 2 changes
#10 main change
#12 move to subpackage

[jira-importer]

Michael Osipov commented

Master is currently broken.

[jira-importer]

Sylwester Lachiewicz commented

Michael Osipov any plan to review? Maybe include this one with new Maven 3.6?

[jira-importer]

Michael Osipov commented

Will review next week.

[jira-importer]

Karl Heinz Marbaise commented

Sylwester Lachiewicz Can you take a look at it cause I want to start a release within the next few days...?

[jira-importer]

Sylwester Lachiewicz commented

Please go without this fix, I'll try to work on this in next year :D

[jira-importer]

Karl Heinz Marbaise commented

Ok Thanks Sylwester for the feedback.

[jira-importer]

Harald Wellmann commented

Rebased Sylwester's changes in a new PR #30 and fixed the failing tests.

Added a new configuration property aether.artifact.descriptor.threads for the number of threads used for parallel download of POMs.

The default is 5. Setting this to 1 reverts to the previous behaviour.

[jira-importer]

Michael Osipov commented

Fixed with 9d3b324e82d9b934bf51bb8818c83caf776e6faf.

[jira-importer]

Tibor Digana commented

Harald Wellmann
We have to revert this feature in resolver and resolver:1.4.0 in Maven. It should be reviewed in a separate branch.

As an explanation, our Maven builds started failing after resolver:1.4.0 has been applied.
To be unbiased and fully logical, I investigated the issue and the feature on ASF Slack. It was me who found several problems in the code, root cause of hanging Maven and deadlock of thread-resources starvation, problems with thread-unsafe classes, overrided objects across threads, one unclear part in the algorithm. And I also found that the algorithm crashed 11 unit tests in maven-resolver after i removed ThreadPoolExecutor (in a playgrounded non-master branch) and thus made it single-threaded.

I am sorry and not glad to say these things, but I would like you and ASF Maven to make code review of this feature, where every question and line must be clarified. Are you open for attending in such code review?

I may copy paste the text from Slack regarding our findings.
But currently we are responsible to solve our internal problems with failing builds.
The Jenkins builds have been failing over one month!

[jira-importer]

Michael Osipov commented

This change has been reverted and moved to the branch pre-MRESOLVER-92.

[jira-importer]

Konrad Windszus commented

Michael Osipov Does MRESOLVER-123 fix some of the issues which appeared with #30?

[jira-importer]

Michael Osipov commented

Konrad Windszus, to be honest, no idea. Maybe, you can try yourself. This is currently out of scope for me.