From 009c5185427aace3d86f3964286e746e442d38a2 Mon Sep 17 00:00:00 2001 From: Tamas Cservenak Date: Wed, 22 Jun 2022 19:32:26 +0200 Subject: [PATCH 1/9] Add some TLP doco about shared local repo access --- src/site/markdown/local-repository.md | 67 +++++++++++++++++---------- 1 file changed, 43 insertions(+), 24 deletions(-) diff --git a/src/site/markdown/local-repository.md b/src/site/markdown/local-repository.md index 9ee50e2d0..0d1062d6e 100644 --- a/src/site/markdown/local-repository.md +++ b/src/site/markdown/local-repository.md @@ -25,28 +25,14 @@ remote, but also to store the artifacts locally installed (locally built and installed, to be more precise). Both of these artifacts were stored in bulk in the local repository. +## Implementations + Local repository implementations implement the `LocalRepositoryManager` (LRM) interface, and Resolver out of the box provides two implementations for it: -"simple" and "enhanced". - -## Simple LRM - -Simple is a fully functional LRM implementation, but is used -mainly in tests, it is not recommended in production environments. - -To manually instantiate a simple LRM, one needs to invoke following code: - -```java -LocalRepositoryManager simple = new SimpleLocalRepositoryManagerFactory() - .newInstance( session, new LocalRepository( baseDir ) ); -``` +"enhanced" used at runtime and "simple" meant to be used in tests and alike +scenarios (is not meant for production use). -Note: This code snippet above instantiates a component, that is not -recommended way to use it, as it should be rather injected whenever possible. -This example above is merely a showcase how to obtain LRM implementation -in unit tests. - -## Enhanced LRM +### Enhanced LRM Enhanced LRM on the other hand is enhanced with several extra features, one most notable is scoping cached content by its origin and context: @@ -58,7 +44,7 @@ Those two, originating from two different repositories may not be the same thing This is meant to protect users from "bad practice" (artifact coordinates are unique in ideal world). -### Split Local Repository +#### Split Local Repository Latest addition to the enhanced LRM is *split* feature. By default, split feature is **not enabled**, enhanced LRM behaves as it behaved in all @@ -75,7 +61,7 @@ The split feature is implemented by the `LocalPathPrefixComposer` interface, that adds different "prefixes" for the locally stored artifacts, based on their context. -#### Note About Release And Snapshot Differentiation +##### Note About Release And Snapshot Differentiation The prefix composer is able to differentiate between release and snapshot versioned artifacts, and this is clear-cut: Maven Artifacts are either @@ -99,7 +85,7 @@ The GAV level metadata gets differentiated based on version it carries, so they may end up in releases or snapshots, depending on their value of `metadata/version` field. -#### Use Cases +##### Use Cases Most direct use case is simpler local repository eviction. One can delete all locally built artifacts without deleting the cached ones, hence, no @@ -139,7 +125,7 @@ $ mvn ... -Daether.enhancedLocalRepository.split \ For complete reference of enhanced LRM configuration possibilities, refer to [configuration page](configuration.html). -#### Split Repository Considerations +##### Split Repository Considerations **Word of warning**: on every change of "split" parameters, user must be aware of the consequences. For example, if one change all aspects of split @@ -149,7 +135,7 @@ is unchanged! Simply put, as all prefixes will be "new", the composed paths will point to potentially non-existing locations, hence, resolver will consider it as a "new" local repository in every aspect. -#### Implementing Custom Split Strategy +##### Implementing Custom Split Strategy To implement custom split strategy, one needs to create a component of type `LocalPathPrefixComposerFactory` and override the default component @@ -160,3 +146,36 @@ class that provides all the defaults. The factory should create a stateless instance of a composer configured from passed in session, that will be used with the enhanced LRM throughout the session. + +### Simple LRM + +Simple is a fully functional LRM implementation, but is used +mainly in tests, it is not recommended in production environments. + +To manually instantiate a simple LRM, one needs to invoke following code: + +```java +LocalRepositoryManager simple = new SimpleLocalRepositoryManagerFactory() + .newInstance( session, new LocalRepository( baseDir ) ); +``` + +Note: This code snippet above instantiates a component, that is not +recommended way to use it, as it should be rather injected whenever possible. +This example above is merely a showcase how to obtain LRM implementation +in unit tests. + +## Shared access to Local Repository + +In case of shared (multi-threaded, multi-process or even multi host) access +to local repository coordination is required, as local repository is hosted +on file system, and each thread may read and write concurrently into it, +causing other threads get incomplete or partially written data. + +Hence, since Resolver 1.7.x line, there is a pluggable API called "Named Locks" +available, providing out of the box lock implementations for cases: + +* multi-threaded, in JVM locking (the default) +* multi-process locking using file system advisory locking +* multi-host locking using Hazelcast or Redisson (needs Redisson or Hazelcast cluster) + +For details see [Named Locks module](maven-resolver-named-locks/). \ No newline at end of file From dac6cd9536e4ab518c97ddced4cf2b897a4e6dbc Mon Sep 17 00:00:00 2001 From: Tamas Cservenak Date: Wed, 22 Jun 2022 19:37:02 +0200 Subject: [PATCH 2/9] Remove contxt --- src/site/markdown/local-repository.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/site/markdown/local-repository.md b/src/site/markdown/local-repository.md index 0d1062d6e..1eda8e721 100644 --- a/src/site/markdown/local-repository.md +++ b/src/site/markdown/local-repository.md @@ -34,7 +34,7 @@ scenarios (is not meant for production use). ### Enhanced LRM -Enhanced LRM on the other hand is enhanced with several extra +Enhanced LRM is enhanced with several extra features, one most notable is scoping cached content by its origin and context: if you downloaded an artifact A1 from repository R1 and later initiate build that requires same artifact A1, but repository R1 From b0bab8ad5df94ffbf5f4b39fcddac17008080baf Mon Sep 17 00:00:00 2001 From: Tamas Cservenak Date: Wed, 22 Jun 2022 19:59:08 +0200 Subject: [PATCH 3/9] Expand named-locks doco --- .../src/site/markdown/index.md.vm | 17 +++++++++++++---- src/site/markdown/local-repository.md | 10 +++++----- 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/maven-resolver-named-locks/src/site/markdown/index.md.vm b/maven-resolver-named-locks/src/site/markdown/index.md.vm index 0d8d40166..63df47771 100644 --- a/maven-resolver-named-locks/src/site/markdown/index.md.vm +++ b/maven-resolver-named-locks/src/site/markdown/index.md.vm @@ -25,17 +25,27 @@ then you can use named locks to make sure they are being protected from concurre Named locks provide support classes for implementations, and provide out of the box seven named lock implementations and three name mappers. -Out of the box, "local" (local to JVM) named lock implementations are the following: +Following implementations are "local" (local to JVM) named lock implementations: - `rwlock-local` implemented in `org.eclipse.aether.named.providers.LocalReadWriteLockNamedLockFactory` that uses JVM `java.util.concurrent.locks.ReentrantReadWriteLock`. - `semaphore-local` implemented in `org.eclipse.aether.named.providers.LocalSemaphoreNamedLockFactory` that uses JVM `java.util.concurrent.Semaphore`. +- `noop` implemented in `org.eclipse.aether.named.providers.NoopNamedLockFactory` that uses no locking. + +Following named lock implementations use underlying OS advisory file locking: + - `file-lock` implemented in `org.eclipse.aether.named.providers.FileLockNamedLockFactory` that uses JVM `java.nio.channels.FileLock`. -- `noop` implemented in `org.eclipse.aether.named.providers.NoopNamedLockFactory` that uses no locking. -Out of the box, "distributed" named lock implementations are the following (separate modules which require additional dependencies): +Note about `file-lock` implementation: it uses OS advisory file locking, hence, concurrently running Maven processes +all set up to use `file-loxk` implementation can safely share one local repository. This is almost certain on +local file systems across all operating systems. In case of NFS mounts, file advisory locking MAY work if NFSv4+ +used with complete setup (with all the necessary services like `RPC` and `portmapper` needed to implement advisory +file locking, check your NFS and/or OS manuals for details). In short: if your (local or remote) FS correctly support +and implements advisory locking, it should work. Local FS usually does, while with NFS your mileage may vary. + +Finally, "distributed" named lock implementations are the following (separate modules which require additional dependencies and remote services): - `rwlock-redisson` implemented in `org.eclipse.aether.named.redisson.RedissonReadWriteLockNamedLockFactory`. - `semaphore-redisson` implemented in `org.eclipse.aether.named.redisson.RedissonSemaphoreNamedLockFactory`. @@ -45,7 +55,6 @@ Out of the box, "distributed" named lock implementations are the following (sepa Local named locks are only suited within one JVM with a multithreaded build. Sharing a local repository between multiple Maven processes (i.e., on a busy CI server) requires a distributed named lock! - The aforementioned (opaque) IDs need to be mapped from artifacts and metadata. Out of the box, name mapper implementations are the following: diff --git a/src/site/markdown/local-repository.md b/src/site/markdown/local-repository.md index 1eda8e721..d360817b5 100644 --- a/src/site/markdown/local-repository.md +++ b/src/site/markdown/local-repository.md @@ -167,15 +167,15 @@ in unit tests. ## Shared access to Local Repository In case of shared (multi-threaded, multi-process or even multi host) access -to local repository coordination is required, as local repository is hosted +to local repository, coordination is a must, as local repository is hosted on file system, and each thread may read and write concurrently into it, -causing other threads get incomplete or partially written data. +causing other threads to get incomplete or partially written data. -Hence, since Resolver 1.7.x line, there is a pluggable API called "Named Locks" -available, providing out of the box lock implementations for cases: +Hence, since Resolver 1.7.x version, there is a pluggable API called "Named Locks" +available, providing out of the box lock implementations for cases like: * multi-threaded, in JVM locking (the default) * multi-process locking using file system advisory locking -* multi-host locking using Hazelcast or Redisson (needs Redisson or Hazelcast cluster) +* multi-host locking using Hazelcast or Redisson (needs Hazelcast or Redisson cluster) For details see [Named Locks module](maven-resolver-named-locks/). \ No newline at end of file From a868bba2897f266afd529020b509a337983e682a Mon Sep 17 00:00:00 2001 From: Tamas Cservenak Date: Wed, 22 Jun 2022 20:03:52 +0200 Subject: [PATCH 4/9] More restructuring --- maven-resolver-named-locks/src/site/markdown/index.md.vm | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/maven-resolver-named-locks/src/site/markdown/index.md.vm b/maven-resolver-named-locks/src/site/markdown/index.md.vm index 63df47771..b192defdd 100644 --- a/maven-resolver-named-locks/src/site/markdown/index.md.vm +++ b/maven-resolver-named-locks/src/site/markdown/index.md.vm @@ -33,6 +33,10 @@ Following implementations are "local" (local to JVM) named lock implementations: JVM `java.util.concurrent.Semaphore`. - `noop` implemented in `org.eclipse.aether.named.providers.NoopNamedLockFactory` that uses no locking. +Note about "local" locks: they are in-JVM, in a way, they properly coordinate in case of multithreaded access from +same JVM, but does not cover accesses across multiple processes and multiple hosts access. +In other words, local named locks are only suited within one JVM with a multithreaded build. + Following named lock implementations use underlying OS advisory file locking: - `file-lock` implemented in `org.eclipse.aether.named.providers.FileLockNamedLockFactory` that uses @@ -52,8 +56,7 @@ Finally, "distributed" named lock implementations are the following (separate mo - `semaphore-hazelcast-client` implemented in `org.eclipse.aether.named.hazelcast.HazelcastClientCPSemaphoreNamedLockFactory`. - `semaphore-hazelcast` implemented in `org.eclipse.aether.named.hazelcast.HazelcastCPSemaphoreNamedLockFactory`. -Local named locks are only suited within one JVM with a multithreaded build. -Sharing a local repository between multiple Maven processes (i.e., on a busy CI server) requires a distributed named lock! +Sharing a local repository between multiple hosts (i.e., on a busy CI server) requires a distributed named lock! The aforementioned (opaque) IDs need to be mapped from artifacts and metadata. From cc8ffc28c6abff1418228d5de073d01ab3bb37ee Mon Sep 17 00:00:00 2001 From: Tamas Cservenak Date: Wed, 22 Jun 2022 20:07:08 +0200 Subject: [PATCH 5/9] Be more specific --- src/site/markdown/local-repository.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/site/markdown/local-repository.md b/src/site/markdown/local-repository.md index d360817b5..b46fa88d5 100644 --- a/src/site/markdown/local-repository.md +++ b/src/site/markdown/local-repository.md @@ -169,7 +169,7 @@ in unit tests. In case of shared (multi-threaded, multi-process or even multi host) access to local repository, coordination is a must, as local repository is hosted on file system, and each thread may read and write concurrently into it, -causing other threads to get incomplete or partially written data. +causing other threads or processes to get incomplete or partially written data. Hence, since Resolver 1.7.x version, there is a pluggable API called "Named Locks" available, providing out of the box lock implementations for cases like: From 12886f0faa5707c0916a68876439deccbc764a8d Mon Sep 17 00:00:00 2001 From: Tamas Cservenak Date: Wed, 22 Jun 2022 20:12:20 +0200 Subject: [PATCH 6/9] Less fuss, more detais --- .../src/site/markdown/index.md.vm | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/maven-resolver-named-locks/src/site/markdown/index.md.vm b/maven-resolver-named-locks/src/site/markdown/index.md.vm index b192defdd..008957508 100644 --- a/maven-resolver-named-locks/src/site/markdown/index.md.vm +++ b/maven-resolver-named-locks/src/site/markdown/index.md.vm @@ -23,7 +23,7 @@ Named locks are essentially locks that are assigned to some given (opaque) ID. I resources that each can have unique ID assigned (i.e., file with an absolute path, some entities with unique ID), then you can use named locks to make sure they are being protected from concurrent read and write actions. -Named locks provide support classes for implementations, and provide out of the box seven named lock implementations and three name mappers. +Named locks provide support classes for implementations, and provide out of the box several lock and name mapper implementations. Following implementations are "local" (local to JVM) named lock implementations: @@ -35,17 +35,17 @@ Following implementations are "local" (local to JVM) named lock implementations: Note about "local" locks: they are in-JVM, in a way, they properly coordinate in case of multithreaded access from same JVM, but does not cover accesses across multiple processes and multiple hosts access. -In other words, local named locks are only suited within one JVM with a multithreaded build. +In other words, local named locks are only suited within one JVM with a multithreaded access. Following named lock implementations use underlying OS advisory file locking: - `file-lock` implemented in `org.eclipse.aether.named.providers.FileLockNamedLockFactory` that uses JVM `java.nio.channels.FileLock`. -Note about `file-lock` implementation: it uses OS advisory file locking, hence, concurrently running Maven processes -all set up to use `file-loxk` implementation can safely share one local repository. This is almost certain on +The `file-lock` implementation uses OS advisory file locking, hence, concurrently running Maven processes +set up to use `file-loxk` implementation can safely share one local repository. This is almost certain on local file systems across all operating systems. In case of NFS mounts, file advisory locking MAY work if NFSv4+ -used with complete setup (with all the necessary services like `RPC` and `portmapper` needed to implement advisory +used with complete setup (with all the necessary services like `RPC` and `portmapper` needed to implement NFS advisory file locking, check your NFS and/or OS manuals for details). In short: if your (local or remote) FS correctly support and implements advisory locking, it should work. Local FS usually does, while with NFS your mileage may vary. @@ -66,3 +66,5 @@ Out of the box, name mapper implementations are the following: - `gav` implemented in `org.eclipse.aether.internal.impl.synccontext.named.GAVNameMapper`. - `discriminating` implemented in `org.eclipse.aether.internal.impl.synccontext.named.DiscriminatingNameMapper`. - `file-gav` implemented in `org.eclipse.aether.internal.impl.synccontext.named.FileGAVNameMapper`. + +Note: the `file-gav` name mapper MUST be used with `file-lock` named locking, no other mapper will work with it. \ No newline at end of file From ad29c60e0359654e86238001d1c22591da77e680 Mon Sep 17 00:00:00 2001 From: Tamas Cservenak Date: Wed, 22 Jun 2022 20:15:46 +0200 Subject: [PATCH 7/9] Typo --- maven-resolver-named-locks/src/site/markdown/index.md.vm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/maven-resolver-named-locks/src/site/markdown/index.md.vm b/maven-resolver-named-locks/src/site/markdown/index.md.vm index 008957508..c17e29117 100644 --- a/maven-resolver-named-locks/src/site/markdown/index.md.vm +++ b/maven-resolver-named-locks/src/site/markdown/index.md.vm @@ -43,7 +43,7 @@ Following named lock implementations use underlying OS advisory file locking: JVM `java.nio.channels.FileLock`. The `file-lock` implementation uses OS advisory file locking, hence, concurrently running Maven processes -set up to use `file-loxk` implementation can safely share one local repository. This is almost certain on +set up to use `file-lock` implementation can safely share one local repository. This is almost certain on local file systems across all operating systems. In case of NFS mounts, file advisory locking MAY work if NFSv4+ used with complete setup (with all the necessary services like `RPC` and `portmapper` needed to implement NFS advisory file locking, check your NFS and/or OS manuals for details). In short: if your (local or remote) FS correctly support From 1f246e8c5ad43666805f6b5407cbc52b6bef09b1 Mon Sep 17 00:00:00 2001 From: Tamas Cservenak Date: Fri, 24 Jun 2022 09:47:56 +0200 Subject: [PATCH 8/9] Weaked the "requires". --- maven-resolver-named-locks/src/site/markdown/index.md.vm | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/maven-resolver-named-locks/src/site/markdown/index.md.vm b/maven-resolver-named-locks/src/site/markdown/index.md.vm index c17e29117..a7e84a00e 100644 --- a/maven-resolver-named-locks/src/site/markdown/index.md.vm +++ b/maven-resolver-named-locks/src/site/markdown/index.md.vm @@ -56,7 +56,8 @@ Finally, "distributed" named lock implementations are the following (separate mo - `semaphore-hazelcast-client` implemented in `org.eclipse.aether.named.hazelcast.HazelcastClientCPSemaphoreNamedLockFactory`. - `semaphore-hazelcast` implemented in `org.eclipse.aether.named.hazelcast.HazelcastCPSemaphoreNamedLockFactory`. -Sharing a local repository between multiple hosts (i.e., on a busy CI server) requires a distributed named lock! +Sharing a local repository between multiple hosts (i.e., on a busy CI server) may be best done with one of distributed named lock, +if NFS locking is not working for you. The aforementioned (opaque) IDs need to be mapped from artifacts and metadata. @@ -67,4 +68,4 @@ Out of the box, name mapper implementations are the following: - `discriminating` implemented in `org.eclipse.aether.internal.impl.synccontext.named.DiscriminatingNameMapper`. - `file-gav` implemented in `org.eclipse.aether.internal.impl.synccontext.named.FileGAVNameMapper`. -Note: the `file-gav` name mapper MUST be used with `file-lock` named locking, no other mapper will work with it. \ No newline at end of file +Note: the `file-gav` name mapper MUST be used with `file-lock` named locking, no other mapper will work with it. From 2a552e3c717e990e3a32e00770c7b69dfd88553d Mon Sep 17 00:00:00 2001 From: Tamas Cservenak Date: Fri, 24 Jun 2022 21:16:16 +0200 Subject: [PATCH 9/9] PR comments --- maven-resolver-named-locks/src/site/markdown/index.md.vm | 6 +++--- src/site/markdown/local-repository.md | 5 +++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/maven-resolver-named-locks/src/site/markdown/index.md.vm b/maven-resolver-named-locks/src/site/markdown/index.md.vm index a7e84a00e..24ed4760b 100644 --- a/maven-resolver-named-locks/src/site/markdown/index.md.vm +++ b/maven-resolver-named-locks/src/site/markdown/index.md.vm @@ -34,15 +34,15 @@ Following implementations are "local" (local to JVM) named lock implementations: - `noop` implemented in `org.eclipse.aether.named.providers.NoopNamedLockFactory` that uses no locking. Note about "local" locks: they are in-JVM, in a way, they properly coordinate in case of multithreaded access from -same JVM, but does not cover accesses across multiple processes and multiple hosts access. +same JVM, but do not cover accesses across multiple processes and/or multiple hosts access. In other words, local named locks are only suited within one JVM with a multithreaded access. -Following named lock implementations use underlying OS advisory file locking: +Following named lock implementations use underlying file system advisory file locking: - `file-lock` implemented in `org.eclipse.aether.named.providers.FileLockNamedLockFactory` that uses JVM `java.nio.channels.FileLock`. -The `file-lock` implementation uses OS advisory file locking, hence, concurrently running Maven processes +The `file-lock` implementation uses file system advisory file locking, hence, concurrently running Maven processes set up to use `file-lock` implementation can safely share one local repository. This is almost certain on local file systems across all operating systems. In case of NFS mounts, file advisory locking MAY work if NFSv4+ used with complete setup (with all the necessary services like `RPC` and `portmapper` needed to implement NFS advisory diff --git a/src/site/markdown/local-repository.md b/src/site/markdown/local-repository.md index b46fa88d5..3554c3962 100644 --- a/src/site/markdown/local-repository.md +++ b/src/site/markdown/local-repository.md @@ -164,7 +164,7 @@ recommended way to use it, as it should be rather injected whenever possible. This example above is merely a showcase how to obtain LRM implementation in unit tests. -## Shared access to Local Repository +## Shared Access to Local Repository In case of shared (multi-threaded, multi-process or even multi host) access to local repository, coordination is a must, as local repository is hosted @@ -178,4 +178,5 @@ available, providing out of the box lock implementations for cases like: * multi-process locking using file system advisory locking * multi-host locking using Hazelcast or Redisson (needs Hazelcast or Redisson cluster) -For details see [Named Locks module](maven-resolver-named-locks/). \ No newline at end of file +For details see [Named Locks module](maven-resolver-named-locks/). +