Skip to content

Commit

Permalink
First recipe: amplify test cases related to code changes
Browse files Browse the repository at this point in the history
  • Loading branch information
danzone committed Apr 19, 2019
1 parent 1461113 commit 2cf9a83
Show file tree
Hide file tree
Showing 3 changed files with 95 additions and 1 deletion.
96 changes: 95 additions & 1 deletion stamp-jenkins-cookbooks/README.md
Original file line number Diff line number Diff line change
@@ -1 +1,95 @@
stamp-jenkins-cookbooks
# STAMP CI/CD Cookbook
This cookbook is for developers, software architects, build managers and QA engineers who want to introduce STAMP test amplification within their CI/CD infrastructure.
A basic understanding of STAMP tools and Jenkins server configuration and usage is expected.

- STAMP tools
- Assess your unit tests with [Descartes](https://github.com/STAMP-project/pitest-descartes) extreme mutation testing
- Amplify your unit tests with [DSpot](https://github.com/stamp-project/dspot)
- Amplify your test configurations with [CAMP](https://github.com/stamp-project/camp)
- Reproduce automatically your production crashes with [Botsing](https://github.com/STAMP-project/botsing/)
- Jenkins
- [Guided tour](https://jenkins.io/doc/pipeline/tour/getting-started/)
- [Jenkins pipelines](https://jenkins.io/doc/book/pipeline/)
- [Blue Ocean](https://jenkins.io/doc/book/blueocean/) Jenkins User Experience


## Table Of Content
- [Amplify unit tests bound to code changes](#amplify-on-code-changes)

## Amplify on code changes
This recipe explains how to use in a combined way [DSpot Diff Selector](https://github.com/STAMP-project/dspot/tree/master/dspot-diff-test-selection) (a tool able to select only unit tests related code changes) and DSpot itself to amplify just those selected unit tests within the CI, leveraging a Jenkins pipeline and [STAMP Pipeline Library](https://github.com/STAMP-project/pipeline-library).
### Getting ready
The pipeline we are going to setup has this structure:

![Pipeline for test amplification on code changes](src/common/images/blue-ocean-pipeline.png "Pipeline for test amplification on code changes")

After compilation and unit test steps, this pipeline will assess your test cases, then in "Amplify" step will retrieve all test cases related to code changes bound to current commit, and will amplify them.

### How to do it
First of all you have to configure Jenkins to use STAMP pipeline code. You can configure it at global level or at build level. Following a screenshot that shows you how to configure it at global level:

![STAMP pipeline library configuration at global level](src/common/images/jenkins-pipeline-config.png "STAMP pipeline library configuration at global level")
Then you have to configure the pipeline in your project repository

Following the pipeline code:

```
@Library('stamp') _
pipeline {
agent any
stages {
stage('Compile') {

This comment has been minimized.

Copy link
@vmassol

vmassol Jun 3, 2019

Shouldn't there be a scm checkout performed before we can build with maven? Maybe this is something automatic with declarative pipelines? (I've never used them, always use scripted pipelines ;)).

steps {
withMaven(maven: 'maven3', jdk: 'JDK8') {
sh "mvn clean compile"
}
}
}
stage('Unit Test') {
steps {
withMaven(maven: 'maven3', jdk: 'JDK8') {
sh "mvn test"
}
}
}
stage ('Test your tests'){
steps {
withMaven(maven: 'maven3', jdk: 'JDK8') {
sh "mvn eu.stamp-project:pitmp-maven-plugin:1.3.6:descartes -DoutputFormats=HTML"
}
publishHTML (target: [
allowMissing: false,
alwaysLinkToLastBuild: false,
keepAll: true,
reportDir: 'target/pit-reports',
reportFiles: '**/index.html',
reportName: "Pit Decartes"
])
}
}
stage('Amplify') {
steps {
script {
stamp.cloneLastStableVersion("oldVersion")

This comment has been minimized.

Copy link
@vmassol

vmassol Jun 3, 2019

Note that this strategy won't work for us since cloneLastStableVersion() find the last commit that had a successful build and we almost never have a successful build ;) (see https://ci.xwiki.org/job/XWiki/job/xwiki-platform/job/master/). Ideally we should get the commit for which the impacted tests last passed. This would mean finding tests and checking test results which is hard to do. Second best is probably to simply use the last commit.

This comment has been minimized.

Copy link
@vmassol

vmassol Jun 3, 2019

Note that this is also costly since it means checkout the full repo which takes up to 10 minutes and several hundreds of MB for xwiki-platform ;)

Maybe we could rely on the already checked out sources and checkout the commit in different directory (from the already checked out sources)?

if (fileExists("${WORKSPACE}/oldVersion/src")){

This comment has been minimized.

Copy link
@vmassol

vmassol May 2, 2019

Are you sure that this will work? AFAIK it'll work only if the github repo being checked out contains a "src" directory at the top level, which is not the case for multi-module Maven projects such as xwiki for example.

This comment has been minimized.

Copy link
@vmassol

vmassol May 2, 2019

I understand it's just an example though. Maybe you can just check for the "oldVersion" directory to be present to be more generic?

This comment has been minimized.

Copy link
@nicolabertazzo

nicolabertazzo May 2, 2019

Contributor

We chose a simple example project so it is easiest to write a cookbook on a simple single maven project (not multi modules).

Instead of src we can check the existence of pom.xml or check that oldVersion dir is not empty.

withMaven(maven: 'maven3', jdk: 'JDK8') {
sh "mvn clean eu.stamp-project:dspot-diff-test-selection:list -Dpath-dir-second-version=${WORKSPACE}/oldVersion"
sh "mvn eu.stamp-project:dspot-maven:amplify-unit-tests -Dpath-to-test-list-csv=testsThatExecuteTheChange.csv -Dverbose -Dtest-criterion=ChangeDetectorSelector -Dpath-to-properties=tavern.properties -Damplifiers=NumberLiteralAmplifier -Diteration=2"

This comment has been minimized.

Copy link
@vmassol

vmassol May 2, 2019

what is this file testsThatExecuteTheChange.csv? Does it need to be provided?

This comment has been minimized.

Copy link
@vmassol

vmassol May 2, 2019

How do you specify a dspot properties file when in a multi-module project?

This comment has been minimized.

Copy link
@nicolabertazzo

nicolabertazzo May 2, 2019

Contributor

You can find documentation about of testsThatExecuteTheChange.csv on dspot-diff documentation: https://github.com/STAMP-project/dspot/tree/master/dspot-diff-test-selection#running-example

In the next version of dspot is not neccesary to specify the dspot properties path (STAMP-project/dspot#768 (comment))

}
}
}
}
}
}
environment {
GIT_URL = sh (script: 'git config remote.origin.url', returnStdout: true).trim().replaceAll('https://','')
}
}
```

The first line imports STAMP Pipeline library.
The stage `Amplify` uses STAMP pipeline library to retrieve the diff cloning the previous last successfull build, and storing it in a folder. Then DSpot Diff Selector will use it to retrieve involved test cases, and then they will be passed to DSpot for the amplification.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 2cf9a83

Please sign in to comment.