[MNG-7433] [REGRESSION] Multiple maven instances working on same source tree can lock each other

[jira-importer]

Dan Tran opened MNG-7433 and commented

I have a large multi modules java maven build where:

• phase 1 - basic build + unit tests + jacoco - 40 min
• phase 2 - sonar:sonar 20 min
• phase 3 - final packaging and basic smoke-test - 20 min

To take advantage of Maven multi-threaded build, during the reactor build, one of our maven module spins another instance of Maven to run sonar:sonar goal right after the basic build is done.

This means our phase 2 and phase 3 run in parallel sharing the same source tree, same local maven repo (where sonar:sonar should have all needed dependencies at the share local maven repo to run its task)

With maven-3.8.5, parallelization is no longer possible, phase 3 is blocked until phase 2 is done.

I am able to trace it to #628 where the locking started the happen

How does the lock mechanic work? there must be a local file where both Maven instances are watching each other. Is there an option to disable this lock?

---

Affects: 3.8.5

Issue Links:

• MNG-7487 Fix deadlock during forked lifecycle executions
  ("is fixed by")
• MNG-7476 Display a warning when an aggregator mojo is locking other mojo executions

Remote Links:

• GitHub Pull Request #162
  
• GitHub Pull Request #729
  
• GitHub Pull Request #730
  

Backported to: 4.0.0-alpha-2, 3.9.0, 3.8.6

0 votes, 9 watchers

[jira-importer]

Michael Osipov commented

gnodet

[jira-importer]

Michael Osipov commented

This doesn't make sense. There is no locking on file level between Maven, BUT can you try Maven from maven-3.9.x? This might be some interaction between this new code an Resolver. 3.9.x contains a dramatically improved Resolver in locking regard.

[jira-importer]

Dan Tran commented

I also tried:

* switch apache maven-invoker to run sonar:sonar ( instead of internal custom invoker)
* replicate a copy of local repo so that maven-invoker has its own maven local repo
* invoke a different goal such as compile, or just a simple sleep  instead of sonar:sonar
* latest  3.9.0 from source

A workaround is to disable this line https://github.com/apache/maven/blob/master/maven-core/src/main/java/org/apache/maven/lifecycle/internal/MojoExecutor.java#L226 will remove the lock

[jira-importer]

Dan Tran commented

The below comment block in interesting

/**
 * Aggregating mojo executions (possibly) modify all MavenProjects, including those that are currently in use
 * by concurrently running mojo executions. To prevent race conditions, an aggregating execution will block
 * all other executions until finished.
 * We also lock on a given project to forbid a forked lifecycle to be executed concurrently with the project.
 * TODO: ideally, the builder should take care of the ordering in a smarter way
 * TODO: and concurrency issues fixed with MNG-7157
 */
private static class ProjectLock implements AutoCloseable

Wonder if running invoke-plugin consider as a 'aggregated project' and its locks other modules at reactor from running

[jira-importer]

Dan Tran commented

gnodet please advice

[jira-importer]

Tamas Cservenak commented

In that PR there is only in-JVM locking involved, there is no way that PR causes that multiple Maven instances (running in different JVMs) locks up each other. How do you run maven exactly? As "usual" so each maven spins up it's own JVM? What JVM you use? What OS you use?

[jira-importer]

Dan Tran commented

Tamas Cservenak thanks for the advice. In my use case i have 2 MVN each of course has its JVM. SuSE Linux +java11 OpenJDK from Amazon. This is definitely a very weird lockup issue

[jira-importer]

Tamas Cservenak commented

I am on Linux as well (Linux Mint 20.3 + Temurin 11.0.14). Can you create a reproducer, or if project does not matter, at least a sequence how you invoke maven instances that lock up?

[jira-importer]

Tamas Cservenak commented

Are you using takari lifecycle or takari local repo maybe?

[jira-importer]

Dan Tran commented

my .mvn/extensions.xml does have

<extension>
   <groupId>io.takari.maven</groupId>
   <artifactId>takari-smart-builder</artifactId>
   <version>0.6.1</version>
</extension>

but it is not activated during maven build, and getting rid of it also does not fix the blocking issue

So far i don't have any luck creating a simple producer. still digging....

[jira-importer]

Dan Tran commented

here is a brief just to visualize the issue and its steps

10:59:16  [INFO] --- maven-install-plugin:2.5.2:install (default-install) @ xxx-mvnrepo ---
10:59:16  [INFO] 
10:59:16  [INFO] ------------------< com.xxx.xxx.xxx:sonar-analysis >------------------
10:59:16  [INFO] Building sonar-analysis 19.11.0-1-SNAPSHOT                     [538/540]
10:59:16  [INFO] --------------------------------[ pom ]---------------------------------
10:59:16  [INFO] 
10:59:16  [INFO] -----------< com.xxx.xxx.xxx.installer:xxx-puppet-config >------------
10:59:16  [INFO] Building xxx-puppet-config 19.11.0-1-SNAPSHOT                  [539/540]
10:59:16  [INFO] --------------------------------[ rpm ]---------------------------------
10:59:16  [INFO] 
10:59:16  [INFO] --- maven-enforcer-plugin:1.4.1:enforce (default) @ sonar-analysis ---
10:59:16  [INFO] 
10:59:16  [INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (jacoco-all) @ sonar-analysis ---
10:59:16  [INFO] argLine set to -javaagent:/space/jenkins/workspace/xxx_xxx_xxx_PR-4236/.repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=/space/jenkins/workspace/xxx_xxx_xxx_PR-4236/sonar/target/jacoco.exec
10:59:16  [INFO] 
10:59:16  [INFO] --- flatten-maven-plugin:1.2.7:flatten (flatten) @ sonar-analysis ---
10:59:16  [INFO] Generating flattened POM of project com.xxx.xxx.xxx:sonar-analysis:pom:19.11.0-1-SNAPSHOT...
10:59:16  [INFO] 
10:59:16  [INFO] --- jacoco-maven-plugin:0.8.7:report (jacoco-all) @ sonar-analysis ---
10:59:16  [INFO] Skipping JaCoCo execution due to missing execution data file.
10:59:20  [INFO] 
10:59:20  [INFO] --- maven-enforcer-plugin:1.4.1:enforce (default) @ xxx-puppet-config ---
10:59:20  [INFO] 
10:59:20  [INFO] --- mdev-maven-plugin:1.6.3:sonar (quiet-sonar-scan-by-default) @ sonar-analysis ---
10:59:20  [INFO] Executing: /bin/sh -c cd '/space/jenkins/workspace/xxx_xxx_xxx_PR-4236' && '/xxx/build/.m2/wrapper/dists/apache-maven-3.8.5-bin/5sf9ufumnhp6odg6f4svjl54q1/apache-maven-3.8.5/bin/mvn' '-ntp' 'sonar:sonar' '-Dsonar.pullrequest.key=4236' '-Dsonar.pullrequest.branch=personal/trand8/maven-3.8.5-2' '-Dsonar.pullrequest.base=master' '--quiet' '-P!sonar' '-s' '/tmp/settings43566726mvn'. Working directory: /space/jenkins/workspace/xxx_xxx_xxx_PR-4236

it shows - after xxx-mvnrepos is done, 2 modules are then run in parallel: sonar-analysis and xxx-puppet-config. The xxx-puppet-config stalls after sonar-analisys spins another maven instant to run sonar:sonar

Note I can replace sonar:sonar with other goal such as 'compile' and see same blocking issue

[jira-importer]

Tamas Cservenak commented

And what arguments are used to invoke the original build. And now I see, it is not the "other" maven that locks here, but the blocking wait within sonar:sonar, that I guess WAITS for the spawned process to finish...

Also, both mojos you'd run, sonar and puppet-config are {}aggregating mojos{}? As aggregating mojos, when running in parallel, may become loose cannon, and that lock is in place exactly to allow only one aggregating mojo to run at same time.... Also, maven has no idea WHAT mojo would do, it seems it's only aggregating.

[jira-importer]

Dan Tran commented

Thanks for the suggestion regarding 'aggregating mojo', I found the consistent pattern where

* after sonar-analysis module execution invoked mdev:sonar ( aggregating mojo which spins a maven instance to run sonar:sonar), other thread is blocked when it runs any aggregating mojos. 

we have a number of internal mojos with 'aggregator = true' set at @Mojo annotation used during the build process.
We set it this way so that it is not running in reactor mode in default

My guess here is I need to update all internal mojos with aggregator=false. btw buildnumber-m-p from MojoHaus is also in the same category

is it something the Maven team can fix in 3.9?

[jira-importer]

Dan Tran commented

Tamas Cservenak from your explanation, can I assume that all 'aggregating' mojo executions are serialized in a multi-threaded reactor build? and it has nothing to do with another Maven instant.

so far I am not able to reproduce it by using 'aggregating' sleep mojo in a multi modules settings

[jira-importer]

Michael Osipov commented

Here is the list of commits. maven-3.8.4...maven-3.8.5
I'd recommend to remove offending commits and see whether it makes a different. But I agree with Tamas Cservenak, there is no way that two instances can lock up each either because all locks are in-JVM.