Skip to content

Extract the caching system into its own module #606

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gnodet wants to merge 18 commits into from
Closed

Extract the caching system into its own module #606

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
dc0928f
[MNG-7324] first step to use Modello for cache-domain
hboutemy Oct 17, 2021
fe5e0ff
[MNG-7324] Complete modello switch
gnodet Nov 4, 2021
a20cb86
[MNG-7324] Use plurals in the schema, make sure it can read things
gnodet Nov 5, 2021
896eeb9
[MNG-7324] Allow more minimal config
gnodet Nov 5, 2021
7159691
[MNG-7324] remove now unused JAXB dependencies
hboutemy Nov 6, 2021
b9985c8
[MNG-7324] update documentation
hboutemy Nov 8, 2021
6047eda
[MNG-7324] split config/domain/diff/report to separate xml.* packages
hboutemy Nov 8, 2021
9c57048
[MNG-7324] rename domain to buildinfo and diff to buildsdiff
hboutemy Nov 8, 2021
cfe213c
[MNG-7321] Move caching in its own module
gnodet Nov 8, 2021
a28f24c
[MNG-7324] Use plural in lists, fix model
gnodet Nov 9, 2021
a7294fa
[MNG-7321] Include the new module in the distribution, uses CDI for a…
gnodet Nov 9, 2021
58cc8ab
[MNG-7322] Use maven session / project properties instead of System.g…
gnodet Nov 9, 2021
9522eb7
[MNG-7321] Use a provided scope for maven-core
gnodet Nov 9, 2021
f078a9a
[MNG-7325] Rename namespaces, add more test for XmlService
gnodet Nov 9, 2021
d86ac76
[MNG-7328] Remove the `Type` suffix on the generated model classes
gnodet Nov 9, 2021
41e7993
[MNG-7324] Update doc regarding model changes
gnodet Nov 9, 2021
cdc3b31
[MNG-7328] Fix tests
gnodet Nov 9, 2021
2c4e998
[MNG-7324] Rename buildinfo -> build and buildsdiff -> diff
gnodet Nov 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
184 changes: 100 additions & 84 deletions Documentation/CACHE-HOWTO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,17 @@

### Overview

Cache configuration provides you additional control over incremental maven behavior. Follow it step by step to
Cache configuration provides you additional control over incremental Maven behavior. Follow it step by step to
understand how it works and figure out your optimal config

### Minimal config

Absolutely minimal config which enables incremental maven with local cache
Absolutely minimal config which enables incremental Maven with local cache

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<cache xmlns="org:apache:maven:cache:config:v1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="org:apache:maven:cache:config:v1 cache-config.xsd">
<cache xmlns="http://maven.apache.org/CACHE-CONFIG/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/CACHE-CONFIG/1.0.0 http://maven.apache.org/xsd/cache-config-1.0.0.xsd">

<configuration>
<enabled>true</enabled>
Expand All @@ -48,61 +47,65 @@ Absolutely minimal config which enables incremental maven with local cache
Just add `<remote>` section under `<configuration>`

```xml

<configuration>
<enabled>true</enabled>
<hashAlgorithm>XX</hashAlgorithm>
<remote>
<url>https://yourserver:port</url>
</remote>
</configuration>
<configuration>
<enabled>true</enabled>
<hashAlgorithm>XX</hashAlgorithm>
<remote>
<url>https://yourserver:port</url>
</remote>
</configuration>
```

### Adding more file types to input

Add all the project specific source code files in `<glob>`. Scala in this case:

```xml

<input>
<global>
<glob>{*.java,*.xml,*.properties,*.scala}</glob>
</global>
</input>
<input>
<global>
<glob>{*.java,*.xml,*.properties,*.scala}</glob>
</global>
</input>
```

### Adding source directory for bespoke project layouts

In most of the cases incremental maven will recognize directories automatically by build introspection. If not, you can
In most of the cases incremental Maven will recognize directories automatically by build introspection. If not, you can
add additional directories with `<include>`. Also you can filter out undesirable dirs and files by using exclude tag

```xml

<input>
<global>
<glob>{*.java,*.xml,*.properties,*.scala}</glob>
<include>importantdir/</include>
<exclude>tempfile.out</exclude>
</global>
</input>
<input>
<global>
<glob>{*.java,*.xml,*.properties,*.scala}</glob>
<includes>
<include>importantdir/</include>
</includes>
<excludes>
<exclude>tempfile.out</exclude>
</excludes>
</global>
</input>
```

### Plugin property is env specific (breaks checksum and caching)

Consider to exclude env specific properties:

```xml

<input>
<global>
...
</global>
<plugin artifactId="maven-surefire-plugin">
<effectivePom>
<excludeProperty>argLine</excludeProperty>
</effectivePom>
</plugin>
</input>
<input>
<global>
...
</global>
<plugins>
<plugin artifactId="maven-surefire-plugin">
<effectivePom>
<excludeProperties>
<excludeProperty>argLine</excludeProperty>
</excludeProperties>
</effectivePom>
</plugin>
</plugins>
</input>
```

Implications - builds with different `argLine` will have identical checksum. Validate that is semantically valid.
Expand All @@ -113,74 +116,87 @@ If plugin configuration property points to `somedir` it will be scanned with def
processing rule

```xml

<input>
<global>
...
</global>
<plugin artifactId="protoc-maven-plugin">
<dirScan mode="auto">
<!--<protoBaseDirectory>${basedir}/..</protoBaseDirectory>-->
<tagScanConfig tagName="protoBaseDirectory" recursive="false" glob="{*.proto}"/>
</dirScan>
</plugin>
</input>
<input>
<global>
...
</global>
<plugins>
<plugin artifactId="protoc-maven-plugin">
<dirScan mode="auto">
<!--<protoBaseDirectory>${basedir}/..</protoBaseDirectory>-->
<tagScanConfigs>
<tagScanConfig tagName="protoBaseDirectory" recursive="false" glob="{*.proto}"/>
</tagScanConfigs>
</dirScan>
</plugin>
</plugins>
</input>
```

### Local repository is not updated because `install` is cached

Add `executionControl/runAlways` section

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<cache xmlns="org:apache:maven:cache:config:v1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="org:apache:maven:cache:config:v1 cache-config.xsd">
<configuration>
...
</configuration>
<input>
...
</input>
<executionControl>
<runAlways>
<plugin artifactId="maven-failsafe-plugin"/>
<execution artifactId="maven-dependency-plugin">
<execId>unpack-autoupdate</execId>
</execution>
<goals artifactId="maven-install-plugin">
<goal>install</goal>
</goals>
<plugins>
<plugin artifactId="maven-failsafe-plugin"/>
</plugins>
<executions>
<execution artifactId="maven-dependency-plugin">
<execIds>
<execId>unpack-autoupdate</execId>
</execIds>
</execution>
</executions>
<goalsLists>
<goalsList artifactId="maven-install-plugin">
<goals>
<goal>install</goal>
</goals>
</goalsList>
</goalsLists>
</runAlways>
</executionControl>
</cache>
```

### I occasionally cached build with `-DskipTests=true` and tests do not run now

If you add command line flags to your build, they do not participate in effective pom - maven defers final value
If you add command line flags to your build, they do not participate in effective pom - Maven defers final value
resolution to plugin runtime. To invalidate build if filed value is different in runtime, add reconciliation section
to `executionControl`:

```xml

<executionControl>
<runAlways>
<?xml version="1.0" encoding="UTF-8" ?>
<cache xmlns="http://maven.apache.org/CACHE-CONFIG/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/CACHE-CONFIG/1.0.0 http://maven.apache.org/xsd/cache-config-1.0.0.xsd">
<configuration>
...
</runAlways>
<reconcile>
<plugin artifactId="maven-surefire-plugin" goal="test">
<reconcile propertyName="skip" skipValue="true"/>
<reconcile propertyName="skipExec" skipValue="true"/>
<reconcile propertyName="skipTests" skipValue="true"/>
<reconcile propertyName="testFailureIgnore" skipValue="true"/>
</plugin>
</reconcile>
</executionControl>
</configuration>
<executionControl>
<runAlways>
...
</runAlways>
<reconcile>
<plugins>
<plugin artifactId="maven-surefire-plugin" goal="test">
<reconciles>
<reconcile propertyName="skip" skipValue="true"/>
<reconcile propertyName="skipExec" skipValue="true"/>
<reconcile propertyName="skipTests" skipValue="true"/>
<reconcile propertyName="testFailureIgnore" skipValue="true"/>
</reconciles>
</plugin>
</plugins>
</reconcile>
</executionControl>
</cache>
```

Please notice `skipValue` attribute. It denotes value which forces skipped execution.
Read `propertyName="skipTests" skipValue="true"` as if property skipTests has value true, plugin will skip execution If
you declare such value incremental maven will reuse appropriate full-build though technically they are different, but
you declare such value incremental Maven will reuse appropriate full-build though technically they are different, but
because full-build is better it is safe to reuse

### How to renormalize line endings in working copy after committing .gitattributes (git 2.16+)
Expand All @@ -201,4 +217,4 @@ git reset --hard
### I want to cache interim build and override it later with final version

Solution: set `-Dremote.cache.save.final=true` to nodes which produce final builds. Such builds will not be overridden
and eventually will replace all interim builds
and eventually will replace all interim builds
4 changes: 2 additions & 2 deletions Documentation/CACHE-PARAMETERS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ This documents contains various configuration parameters supported by cache engi
| `-Dremote.cache.configPath=true/false` | Location of cache configuration file | Cache config is not in default location |
| `-Dremote.cache.enabled=true/false` | Remote cache and associated features disabled/enabled | To remove noise from logs then remote cache is not available |
| `-Dremote.cache.save.enabled=true/false` | Remote cache save allowed or not | To designate nodes which allowed to push in remote shared cache |
| `-Dremote.cache.save.final=true/false` | Is it allowed or not to override produced cache | To ensure that reference build is not overriden by interim build |
| `-Dremote.cache.failFast=true/false` | Fail on the first module whcih cannot be restored from cache | Remote cache setup/tuning/troubleshooting |
| `-Dremote.cache.save.final=true/false` | Is it allowed or not to override produced cache | To ensure that reference build is not overridden by interim build |
| `-Dremote.cache.failFast=true/false` | Fail on the first module which cannot be restored from cache | Remote cache setup/tuning/troubleshooting |
| `-Dremote.cache.baselineUrl=<http url>` | Location of baseline build for comparison | Remote cache setup/tuning/troubleshooting |

## Project level properties
Expand Down
8 changes: 4 additions & 4 deletions Documentation/CACHE-REMOTE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

# Overview

This document describes generic approach to cache setup. The process require expertise in maven equivalent to expertise
This document describes generic approach to cache setup. The process require expertise in Maven equivalent to expertise
required to author and Maven your project build, it implies good knowledge of both Maven and the project. Due to Maven
model limitation the process is manual, but allows you to achieve sufficient control and transparency over caching
logic.
Expand Down Expand Up @@ -113,8 +113,8 @@ Once discrepancy between remote and local builds detected cache will fail with d
in `target/incremental-maven` directory:

```
* buildinfo-baseline-3c64673e23259e6f.xml - build specficiation from baseline build
* buildinfo-db43936e0666ce7.xml - build specification of locall build
* buildinfo-baseline-3c64673e23259e6f.xml - build specification from baseline build
* buildinfo-db43936e0666ce7.xml - build specification of local build
* buildsdiff.xml - comparison report with list of discrepancies
```

Expand Down Expand Up @@ -242,7 +242,7 @@ relax consistency rules in favor of compatibility, remove property from tracked

Current implementation doesn't support version changes between cache entries. It will result in cache invalidation for
each new version.
To mitigate the issue please consider migrating off traditional maven release approach - try to use single version id in
To mitigate the issue please consider migrating off traditional Maven release approach - try to use single version id in
project (eg `<version>MY-PROJECT-LOCAL</version>`). Such approach simplifies git branching workflow significantly.

Deployment of artifacts with specific version from builds with cache is not supported yet.
Expand Down
Loading