Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 22 additions & 7 deletions maven-resolver-named-locks/src/site/markdown/index.md.vm
Original file line number Diff line number Diff line change
Expand Up @@ -23,28 +23,41 @@ Named locks are essentially locks that are assigned to some given (opaque) ID. I
resources that each can have unique ID assigned (i.e., file with an absolute path, some entities with unique ID),
then you can use named locks to make sure they are being protected from concurrent read and write actions.

Named locks provide support classes for implementations, and provide out of the box seven named lock implementations and three name mappers.
Named locks provide support classes for implementations, and provide out of the box several lock and name mapper implementations.

Out of the box, "local" (local to JVM) named lock implementations are the following:
Following implementations are "local" (local to JVM) named lock implementations:

- `rwlock-local` implemented in `org.eclipse.aether.named.providers.LocalReadWriteLockNamedLockFactory` that uses
JVM `java.util.concurrent.locks.ReentrantReadWriteLock`.
- `semaphore-local` implemented in `org.eclipse.aether.named.providers.LocalSemaphoreNamedLockFactory` that uses
JVM `java.util.concurrent.Semaphore`.
- `noop` implemented in `org.eclipse.aether.named.providers.NoopNamedLockFactory` that uses no locking.

Note about "local" locks: they are in-JVM, in a way, they properly coordinate in case of multithreaded access from
same JVM, but do not cover accesses across multiple processes and/or multiple hosts access.
In other words, local named locks are only suited within one JVM with a multithreaded access.

Following named lock implementations use underlying file system advisory file locking:

- `file-lock` implemented in `org.eclipse.aether.named.providers.FileLockNamedLockFactory` that uses
JVM `java.nio.channels.FileLock`.
- `noop` implemented in `org.eclipse.aether.named.providers.NoopNamedLockFactory` that uses no locking.

Out of the box, "distributed" named lock implementations are the following (separate modules which require additional dependencies):
The `file-lock` implementation uses file system advisory file locking, hence, concurrently running Maven processes
set up to use `file-lock` implementation can safely share one local repository. This is almost certain on
local file systems across all operating systems. In case of NFS mounts, file advisory locking MAY work if NFSv4+
used with complete setup (with all the necessary services like `RPC` and `portmapper` needed to implement NFS advisory
file locking, check your NFS and/or OS manuals for details). In short: if your (local or remote) FS correctly support
and implements advisory locking, it should work. Local FS usually does, while with NFS your mileage may vary.

Finally, "distributed" named lock implementations are the following (separate modules which require additional dependencies and remote services):

- `rwlock-redisson` implemented in `org.eclipse.aether.named.redisson.RedissonReadWriteLockNamedLockFactory`.
- `semaphore-redisson` implemented in `org.eclipse.aether.named.redisson.RedissonSemaphoreNamedLockFactory`.
- `semaphore-hazelcast-client` implemented in `org.eclipse.aether.named.hazelcast.HazelcastClientCPSemaphoreNamedLockFactory`.
- `semaphore-hazelcast` implemented in `org.eclipse.aether.named.hazelcast.HazelcastCPSemaphoreNamedLockFactory`.

Local named locks are only suited within one JVM with a multithreaded build.
Sharing a local repository between multiple Maven processes (i.e., on a busy CI server) requires a distributed named lock!

Sharing a local repository between multiple hosts (i.e., on a busy CI server) may be best done with one of distributed named lock,
if NFS locking is not working for you.

Copy link
Copy Markdown
Member

@kwin kwin Jun 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the example CI server is not the best as I don’t know of any tools which recommend or ease setting up a shared Maven repo used by distributed nodes…

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, but that's actually the point as well: they do not recommend as Maven had issues with it before resolver 1.7 😉

The aforementioned (opaque) IDs need to be mapped from artifacts and metadata.

Expand All @@ -54,3 +67,5 @@ Out of the box, name mapper implementations are the following:
- `gav` implemented in `org.eclipse.aether.internal.impl.synccontext.named.GAVNameMapper`.
- `discriminating` implemented in `org.eclipse.aether.internal.impl.synccontext.named.DiscriminatingNameMapper`.
- `file-gav` implemented in `org.eclipse.aether.internal.impl.synccontext.named.FileGAVNameMapper`.

Note: the `file-gav` name mapper MUST be used with `file-lock` named locking, no other mapper will work with it.
70 changes: 45 additions & 25 deletions src/site/markdown/local-repository.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,30 +25,16 @@ remote, but also to store the artifacts locally installed (locally built and
installed, to be more precise). Both of these artifacts were stored in bulk
in the local repository.

## Implementations

Local repository implementations implement the `LocalRepositoryManager` (LRM)
interface, and Resolver out of the box provides two implementations for it:
"simple" and "enhanced".

## Simple LRM
"enhanced" used at runtime and "simple" meant to be used in tests and alike
scenarios (is not meant for production use).

Simple is a fully functional LRM implementation, but is used
mainly in tests, it is not recommended in production environments.

To manually instantiate a simple LRM, one needs to invoke following code:
### Enhanced LRM

```java
LocalRepositoryManager simple = new SimpleLocalRepositoryManagerFactory()
.newInstance( session, new LocalRepository( baseDir ) );
```

Note: This code snippet above instantiates a component, that is not
recommended way to use it, as it should be rather injected whenever possible.
This example above is merely a showcase how to obtain LRM implementation
in unit tests.

## Enhanced LRM

Enhanced LRM on the other hand is enhanced with several extra
Enhanced LRM is enhanced with several extra
features, one most notable is scoping cached content by its origin and context:
if you downloaded an artifact A1 from repository R1
and later initiate build that requires same artifact A1, but repository R1
Expand All @@ -58,7 +44,7 @@ Those two, originating from two different repositories may not be the same thing
This is meant to protect users from "bad practice" (artifact coordinates are
unique in ideal world).

### Split Local Repository
#### Split Local Repository

Latest addition to the enhanced LRM is *split* feature. By default, split
feature is **not enabled**, enhanced LRM behaves as it behaved in all
Expand All @@ -75,7 +61,7 @@ The split feature is implemented by the `LocalPathPrefixComposer` interface,
that adds different "prefixes" for the locally stored artifacts, based on
their context.

#### Note About Release And Snapshot Differentiation
##### Note About Release And Snapshot Differentiation

The prefix composer is able to differentiate between release and snapshot
versioned artifacts, and this is clear-cut: Maven Artifacts are either
Expand All @@ -99,7 +85,7 @@ The GAV level metadata gets differentiated based on version it carries, so
they may end up in releases or snapshots, depending on their value of
`metadata/version` field.

#### Use Cases
##### Use Cases

Most direct use case is simpler local repository eviction. One can delete all
locally built artifacts without deleting the cached ones, hence, no
Expand Down Expand Up @@ -139,7 +125,7 @@ $ mvn ... -Daether.enhancedLocalRepository.split \
For complete reference of enhanced LRM configuration possibilities, refer to
[configuration page](configuration.html).

#### Split Repository Considerations
##### Split Repository Considerations

**Word of warning**: on every change of "split" parameters, user must be aware
of the consequences. For example, if one change all aspects of split
Expand All @@ -149,7 +135,7 @@ is unchanged! Simply put, as all prefixes will be "new", the composed paths will
point to potentially non-existing locations, hence, resolver will consider
it as a "new" local repository in every aspect.

#### Implementing Custom Split Strategy
##### Implementing Custom Split Strategy

To implement custom split strategy, one needs to create a component of
type `LocalPathPrefixComposerFactory` and override the default component
Expand All @@ -160,3 +146,37 @@ class that provides all the defaults.
The factory should create a stateless instance of a composer
configured from passed in session, that will be used with the enhanced LRM
throughout the session.

### Simple LRM

Simple is a fully functional LRM implementation, but is used
mainly in tests, it is not recommended in production environments.

To manually instantiate a simple LRM, one needs to invoke following code:

```java
LocalRepositoryManager simple = new SimpleLocalRepositoryManagerFactory()
.newInstance( session, new LocalRepository( baseDir ) );
```

Note: This code snippet above instantiates a component, that is not
recommended way to use it, as it should be rather injected whenever possible.
This example above is merely a showcase how to obtain LRM implementation
in unit tests.

## Shared Access to Local Repository

In case of shared (multi-threaded, multi-process or even multi host) access
to local repository, coordination is a must, as local repository is hosted
on file system, and each thread may read and write concurrently into it,
causing other threads or processes to get incomplete or partially written data.

Hence, since Resolver 1.7.x version, there is a pluggable API called "Named Locks"
available, providing out of the box lock implementations for cases like:

* multi-threaded, in JVM locking (the default)
* multi-process locking using file system advisory locking
* multi-host locking using Hazelcast or Redisson (needs Hazelcast or Redisson cluster)

For details see [Named Locks module](maven-resolver-named-locks/).