Skip to content
This repository has been archived by the owner on Aug 5, 2022. It is now read-only.

Incremental build and shared cache for Apache Maven #1

Open
wants to merge 1 commit into
base: MNG-7129/maven-3.6.3-base
Choose a base branch
from
Open

Incremental build and shared cache for Apache Maven #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ target/
.classpath
.settings/
.svn/
.cache/
bin/
# Intellij
*.ipr
Expand Down
204 changes: 204 additions & 0 deletions Documentation/CACHE-HOWTO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

### Overview

Cache configuration provides you additional control over incremental maven behavior. Follow it step by step to
understand how it works and figure out your optimal config

### Minimal config

Absolutely minimal config which enables incremental maven with local cache

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<cache xmlns="org:apache:maven:cache:config:v1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="org:apache:maven:cache:config:v1 cache-config.xsd">

<configuration>
<enabled>true</enabled>
<hashAlgorithm>XX</hashAlgorithm>
</configuration>

<input>
<global>
<glob>{*.java,*.xml,*.properties}</glob>
</global>
</input>
</cache>
```

### Enabling remote cache

Just add `<remote>` section under `<configuration>`

```xml

<configuration>
<enabled>true</enabled>
<hashAlgorithm>XX</hashAlgorithm>
<remote>
<url>https://yourserver:port</url>
</remote>
</configuration>
```

### Adding more file types to input

Add all the project specific source code files in `<glob>`. Scala in this case:

```xml

<input>
<global>
<glob>{*.java,*.xml,*.properties,*.scala}</glob>
</global>
</input>
```

### Adding source directory for bespoke project layouts

In most of the cases incremental maven will recognize directories automatically by build introspection. If not, you can
add additional directories with `<include>`. Also you can filter out undesirable dirs and files by using exclude tag

```xml

<input>
<global>
<glob>{*.java,*.xml,*.properties,*.scala}</glob>
<include>importantdir/</include>
<exclude>tempfile.out</exclude>
</global>
</input>
```

### Plugin property is env specific (breaks checksum and caching)

Consider to exclude env specific properties:

```xml

<input>
<global>
...
</global>
<plugin artifactId="maven-surefire-plugin">
<effectivePom>
<excludeProperty>argLine</excludeProperty>
</effectivePom>
</plugin>
</input>
```

Implications - builds with different `argLine` will have identical checksum. Validate that is semantically valid.

### Plugin property points to directory where only subset of files is relevant

If plugin configuration property points to `somedir` it will be scanned with default glob. You can tweak it with custom
processing rule

```xml

<input>
<global>
...
</global>
<plugin artifactId="protoc-maven-plugin">
<dirScan mode="auto">
<!--<protoBaseDirectory>${basedir}/..</protoBaseDirectory>-->
<tagScanConfig tagName="protoBaseDirectory" recursive="false" glob="{*.proto}"/>
</dirScan>
</plugin>
</input>
```

### Local repository is not updated because `install` is cached

Add `executionControl/runAlways` section

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<cache xmlns="org:apache:maven:cache:config:v1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="org:apache:maven:cache:config:v1 cache-config.xsd">
<configuration>
...
</configuration>
<input>
...
</input>
<executionControl>
<runAlways>
<plugin artifactId="maven-failsafe-plugin"/>
<execution artifactId="maven-dependency-plugin">
<execId>unpack-autoupdate</execId>
</execution>
<goals artifactId="maven-install-plugin">
<goal>install</goal>
</goals>
</runAlways>
</executionControl>
</cache>
```

### I occasionally cached build with `-DskipTests=true` and tests do not run now

If you add command line flags to your build, they do not participate in effective pom - maven defers final value
resolution to plugin runtime. To invalidate build if filed value is different in runtime, add reconciliation section
to `executionControl`:

```xml

<executionControl>
<runAlways>
...
</runAlways>
<reconcile>
<plugin artifactId="maven-surefire-plugin" goal="test">
<reconcile propertyName="skip" skipValue="true"/>
<reconcile propertyName="skipExec" skipValue="true"/>
<reconcile propertyName="skipTests" skipValue="true"/>
<reconcile propertyName="testFailureIgnore" skipValue="true"/>
</plugin>
</reconcile>
</executionControl>
```

Please notice `skipValue` attribute. It denotes value which forces skipped execution.
Read `propertyName="skipTests" skipValue="true"` as if property skipTests has value true, plugin will skip execution If
you declare such value incremental maven will reuse appropriate full-build though technically they are different, but
because full-build is better it is safe to reuse

### How to renormalize line endings in working copy after committing .gitattributes (git 2.16+)

Ensure you've committed (and ideally pushed everything) - no changes in working copy. After that:

```shell
# Rewrite objects and update index
git add --renormalize .
# Commit changes
git commit -m "Normalizing line endings"
# Remove working copy paths from git cache
git rm --cached -r .
# Refresh with new line endings
git reset --hard
```

### I want to cache interim build and override it later with final version

Solution: set `-Dremote.cache.save.final=true` to nodes which produce final builds. Such builds will not be overridden
and eventually will replace all interim builds
52 changes: 52 additions & 0 deletions Documentation/CACHE-PARAMETERS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# Overview

This documents contains various configuration parameters supported by cache engine

## Command line flags

| Parameter | Description | Usage Scenario |
| ----------- | ----------- | ----------- |
| `-Dremote.cache.configPath=true/false` | Location of cache configuration file | Cache config is not in default location |
| `-Dremote.cache.enabled=true/false` | Remote cache and associated features disabled/enabled | To remove noise from logs then remote cache is not available |
| `-Dremote.cache.save.enabled=true/false` | Remote cache save allowed or not | To designate nodes which allowed to push in remote shared cache |
| `-Dremote.cache.save.final=true/false` | Is it allowed or not to override produced cache | To ensure that reference build is not overriden by interim build |
| `-Dremote.cache.failFast=true/false` | Fail on the first module whcih cannot be restored from cache | Remote cache setup/tuning/troubleshooting |
| `-Dremote.cache.baselineUrl=<http url>` | Location of baseline build for comparison | Remote cache setup/tuning/troubleshooting |

## Project level properties

Project level parameters allow overriding global parameters on project level Must be specified as project properties:

```xml

<pom>
...
<properties>
<remote.cache.input.glob>{*.css}</remote.cache.input.glob>
</properties>
</pom>
```

| Parameter | Description |
| ----------------------------- | ----------- |
| `remote.cache.input.glob` | Project specific glob to select sources. Overrides global glob. |
| `remote.cache.input` | Property prefix to mark paths which must be additionally scanned for source code. Value of property starting with this prefix will be treated as path relatively to current project/module |
| `remote.cache.exclude` | Property prefix to mark paths which must be excluded from source code search. Value of property starting with this prefix will be treated as path to current project/module |
| `remote.cache.processPlugins` | Introspect plugins to find inputs or not. Default is true. |
Loading