real demo

Real Demo (dev in progress)

This demo will show how to prepare a useful MPL pipeline for a kubernetes jenkins environment and support infrastructure (git service, artifact storage).

We will use minikube, gitea, nexus, jenkins, unit and parallel integration tests, common lifecycle dev -> qa -> prod.

Uses a modified petclinic example: https://github.com/deors/deors-demos-petclinic

WARNING!

This is just an example of the infrastructure - it doesn't have a proper security enforcement (but could be easily added).
All the variables we entering manually here are set by default in the preconfiguration - so if you want to change something grep and change the configs first.
Make sure you have enough resources: I executed the example on the instance with 8vCPU and 16GB of RAM.

Setup

Step I: install kubernetes & dashboard

We will use kubernetes to simplify the envs separation and use the better jenkins agents.

Install kubectl & minikube - there is a number of ways, but for MacOS it's better to use Docker Desktop, for linux and for windows: https://kubernetes.io/docs/tasks/tools/install-minikube/
- If you using linux - make sure your /tmp directory is mounted with no noexec. That's needed for PVC's that uses /tmp/hostpath-provisioner directory to store the dirs. Otherwise the applications requires execution on their data (git uses exec hooks in the repos) will fail.

Setup kubernetes dashboard - it's useful for monitoring and showing what's happening on the cluster. Choose the right version for your minikube (install could be different depends on the version you're going to use): https://github.com/kubernetes/dashboard/releases

$ KUBE_DB_VERSION=v2.0.0

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/${KUBE_DB_VERSION}/aio/deploy/recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

Apply the service admin-user account to access dashboard:

$ kubectl apply -f k8s-components/01-dahsboard-access.yaml
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created

Setup the namespaces we will use for our infrastructure and applications:

$ kubectl apply -f k8s-components/02-namespaces.yaml
namespace/example-infra created
namespace/example-dev created
namespace/example-qa created
namespace/example-prod created

Start local proxy to access the kubernetes services:
```
$ kubectl proxy
```
You can login to the k8s dasboard using token from the next command and url: http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
```
$ kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
<output with token will be printed here>
```
Check that the namespaces was created on the dashboard

Ok, now we have a solid base for our system. Next we need to install the components will help us to setup the infrastructure.

Step II: install gitea

This small git server will help us to trigger the jenkins jobs when there some changes will come.

Setup simple gitea git server (downloading of the image could take some time)

$ kubectl apply -f k8s-components/03-gitea-pvc.yaml,k8s-components/03-gitea.yaml
persistentvolumeclaim/gitea-data-pvc created
service/git-service created
statefulset.apps/gitea-app created

Gitea working not well with proxy, so - port-forwarding:

$ kubectl -n example-infra port-forward service/gitea-service 3000:web
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

Configure gitea: go to http://localhost:3000/install
- Select SQLite3 database
- Go to Server and Third-Party Service Settings and select Enable Local Mode
- Go to Administrator Account Settings and create "git-admin" account
- Click Install button

Step III: install nexus

Will store our maven artifacts right after the build.

Install nexus3 artifact server (downloading of the image could take some time)

$ kubectl apply -f k8s-components/04-nexus-pvc.yaml,k8s-components/04-nexus.yaml
persistentvolumeclaim/nexus-data-pvc created
service/nexus-service created
statefulset.apps/nexus-app created

Run the port forwarding (nexus UI don't like proxy, make sure socat is installed on linux)

$ kubectl -n example-infra port-forward service/nexus-service 4000:web
Forwarding from 127.0.0.1:4000 -> 8080
Forwarding from [::1]:4000 -> 8080

Login and setup nexus:
- Go to the nexus web page and click Sign In button: http://localhost:4000/
- Use dashboard Exec into Pod button or docker exec command to cat the /nexus-data/admin.password file on the container
- Change admin password
- Set Enable anonymous access flag
Add jenkinsci repository proxy (we need it to build shared libraries):
- Go to repositories: http://localhost:4000/#admin/repository/repositories
- Click Create repository button
- Select "maven2 (proxy)"
- Set Name "jenkinsci"
- Set Remote storage to "http://repo.jenkins-ci.org/public/"
- Click Create repository button at the bottom
- Go to "maven-public" group: http://localhost:4000/#admin/repository/repositories:maven-public
- Add "jenkinsci" to Members list
- Click Save button at the bottom
Create new role to handle jenkins access:
- Go to roles settings: http://localhost:4000/#admin/security/roles
- Click Create role and select "Nexus role" item
- Set Role ID and Role name to "jenkins-deploy"
- Move Privileges available item "nx-repository-view-*-*-*" to Given field
- Click Create role button to save the new role
Add jenkins user to jenkins-deploy role:
- Go to users settings: http://localhost:4000/#admin/security/users
- Click on Create local user button
- Set ID and Password to "jenkins-deploy"
- Select Status "Active"
- Move role "jenkins-deploy" to Granted
- Set the other req fields as you like
- Click to Create local user button to save the new user

Step IV: install jenkins

We will use jenkinsBro to simplify the installation:

Clone the jenkinsBro repository:

$ git clone https://github.com/rabits/jenkinsbro.git

Modify the plugins.txt list with required plugins for our pipelines:
- config-file-provider - maven global settings
- kubernetes - cloud to run ondemand agents
- gogs-webhook - automatically trigger builds from gitea git changes (we will use it to build MPL)
- pipeline-utility-steps - contains useful parser pipeline steps (for pom files for example)
- custom-tools-plugin - used for custom tools: kaniko
```
$ echo "config-file-provider\nkubernetes\ngogs-webhook\npipeline-utility-steps\ncustom-tools-plugin" >> ./jenkinsbro/plugins.txt
```
Build the image of the latest jenkins LTS - it will generate jenkinsbro-master image:
```
$ ./jenkinsbro/jenkinsbro_build.sh lts
```

Install jenkins to the example-infra namespace (yaml file contains preconfiguration, it's a good idea to read it):

$ kubectl apply -f k8s-components/05-jenkins-pvc.yaml,k8s-components/05-jenkins.yaml
persistentvolumeclaim/jenkins-data-pvc created
service/jenkins-service created
configmap/jenkinsbro-config created
statefulset.apps/jenkins-app created

Step V: build a minimal jnlp agent

We will use slightly advanced kubernetes agent pod - it will be built from 2 containers: jnlp and worker. This is needed for a simple thing - to properly limit the memory consumed by the workload and to make sure agent will not be killed by kubernetes if workload will exceed the memory limit. BTW container will use jnlp agent provided by our jenkins master.

Build the image of jnlp agent available in the real-demo directory:

$ docker build -t jenkins-agent-jnlp k8s-build/jenkins-agent-jnlp

Configuration of simple build

Ok, we prepared the base infrastructure and now need to fill it with some useful stuff. Since this example is about MPL, we will create the nested shared library with our overrides to build the simple MPL self-build example and store it in nexus to use later.

Step I: configure gitea and mpl mirror repository

We will need a mirror of the MPL repository (to make sure internet shutdown will not affect us)

Open gitea: http://localhost:3000/
Create example organization:
- Click on + Create menu
- Select New Organization item
- Set Organization Name to "example-org"
- Click Create Organization button
Create mirror of the MPL repo in the example-org:
- Click on + Create menu
- Select New Migration item
- Set Owner to "example-org"
- Set Clone From URL to "https://github.com/griddynamics/mpl"
- Enable Migration Type flag - we're not going to modify this repo
- Click Migrate Repository button

Step II: create project cicd nested library repository

We can't use the default MPL build in our infrastructure, so we have to create an additional nested shared library with a number of overrides.

Open gitea: http://localhost:3000/
Create new repo to store our example project cicd lib:
- Click on + Create menu
- Select New Repository item
- Set Owner to "example-org"
- Set Repository Name to "example-cicd"
- Click on Create Repository button

Prepare and push the shared lib:

Init git repo in the existing shared library dir

$ git init example-cicd
Initialized empty Git repository in example-cicd/.git/

Commit the existing workspace files

$ git -C example-cicd add .
$ git -C example-cicd commit . -m 'Init'

Run the port forwarding to push the data (make sure socat is installed on linux)

$ kubectl -n example-infra port-forward service/git-service 3000:web
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

Add remote and push the commit

$ git -C example-cicd remote add origin "http://localhost:3000/example-org/example-cicd.git"
$ git -C example-cicd push -u origin master
Handling connection for 3000
Username for 'http://localhost:3000': git-admin
Password for 'http://git-admin@localhost:3000':
Counting objects: 59, done.
Delta compression using up to 48 threads.
Compressing objects: 100% (42/42), done.
Writing objects: 100% (59/59), 13.20 KiB | 1.47 MiB/s, done.
Total 59 (delta 2), reused 0 (delta 0)
To http://localhost:3000/example-org/example-cicd.git
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin' by rebasing.

Step III: create mpl-master jenkins job

Ok, now we're ready to actually setup the pipeline - let's start with minimal MPL build: we need the artifacts stored in nexus to move forward.

Run the port forwarding (make sure socat is installed on linux)

$ kubectl -n example-infra port-forward service/jenkins-service 8089:web
Forwarding from 127.0.0.1:8089 -> 8080
Forwarding from [::1]:8089 -> 8080

Go to Jenkins: http://localhost:8089/

Create new job "mpl-master" type "Pipeline"

Enable Branch Filter and set "master"

In the Pipeline script put the next content:

@Library('example-cicd') _

ExampleProjectPipeline {
  git = [
    url: 'http://git-service/example-org/mpl',
    branch: 'master',
  ]
  // Disabling not needed stages
  modules.Deploy = null
  modules.Test = null
}

Save the job "mpl-master"

Set the gogs webhook:
- Go to gitea mpl repo webhooks settings: http://localhost:3000/example-org/mpl/settings/hooks
- Click Add Webhook, select "Gogs" item
- Set Payload URL to "http://jenkins-service/gogs-webhook/?job=mpl-master"
- Click Add Webhook
Go into the created webhook and click Test Delivery button at the bottom
Go back to the jenkins "mpl-master" job: http://localhost:8085/job/mpl-master/ and see how build is executing

Validating our shared library

The example shared library we uploaded contains a number of tests, so we will compile and execute unit tests for each change of the library. Let's create example-cicd-build jenkins job and setup the required hooks:

Go to Jenkins: http://localhost:8085/
Create new job "example-cicd-build" type "Multibranch Pipeline"
- Add "Git" to the Branch Sources
- Set Project Repository "http://git-service/example-org/example-cicd"
- Click Save button at the bottom - after that multibranch job will check the available branches and start the master build
Set the gitea webhook - it will trigger changes build using the git plugin:
- Go to gitea example-cicd repo webhooks settings: http://localhost:3000/example-org/example-cicd/settings/hooks
- Click Add Webhook, select "Gogs" item
- Set Payload URL to "http://jenkins-service/git/notifyCommit?url=http://git-service/example-org/example-cicd"
- Click Add Webhook

Real deal: petclinic application

Now we need an application that we could build, deploy, test and move through the release process and environments. We will take a slightly modified petclinic application (as we've used in the petclinic-selenium example).

Step I: create new petclinic repository and upload the sources

Create new repo to store our petclinic sources:
- Go into example-org
- Click Create Repository button
- Set Repository Name to "petclinic"
- Click on Create Repository button

Prepare and push petclinic sources:

Init git repo in the existing petclinic dir

$ git init petclinic
Initialized empty Git repository in petclinic/.git/

Commit the existing workspace files

$ git -C petclinic add .
$ git -C petclinic commit . -m 'Init'

Add remote and push the commit

$ git -C petclinic remote add origin http://localhost:3000/example-org/petclinic.git
$ git -C petclinic push -u origin master
Counting objects: 187, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (172/172), done.
Writing objects: 100% (187/187), 176.47 KiB | 4.52 MiB/s, done.
Total 187 (delta 40), reused 0 (delta 0)
remote: Resolving deltas: 100% (40/40), done.
To http://localhost:3000/example-org/petclinic.git
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin' by rebasing.

Step II: prepare a kaniko tool binary

Ok, we have the application, but to pack it to docker image with k8s - we need a special tool named kaniko. It can build docker image without any access to docker daemon, means we can use it in unprevileged docker container (good for security indeed).

Unfortunately it's oficially distributed only as a docker image, so extracting of the tool will take some time.

Create the kaniko container:

$ docker create --name kaniko gcr.io/kaniko-project/executor:v0.19.0
Unable to find image 'gcr.io/kaniko-project/executor:v0.19.0' locally
v0.19.0: Pulling from kaniko-project/executor
c30f0b4c9053: Pull complete
ed36162ea2d3: Pull complete
934aba279703: Pull complete
958c06b88e30: Pull complete
9fa803ac1ec6: Pull complete
03a44a7314d7: Pull complete
861e678c6f4c: Pull complete
e60dcc1bf57d: Pull complete
Digest: sha256:66be3f60f22b571faa82e0aaeb94731217ba0c58ac4a3b062bc84c6d8d545213
Status: Downloaded newer image for gcr.io/kaniko-project/executor:v0.19.0
31b512e80ec112293978b6eca12bd7e193c95e6bbcfb8c80f142e8e724321ab5

Copy the executor to the local fs:

$ docker cp kaniko:/kaniko/executor /tmp/kaniko

Delete the docker container:
```
$ docker rm kaniko
```

Pack the kaniko binary to a zip archive:

$ zip -9 -j /tmp/kaniko.zip /tmp/kaniko

Upload the archive to nexus:

$ curl -u jenkins-deploy:jenkins-deploy --upload-file /tmp/kaniko.zip http://localhost:4000/repository/maven-releases/com.google.GoogleContainerTools/kaniko/0.19.0/kaniko-0.19.0.zip

Make sure the custom tool kaniko is already created in the global tools of Jenkins - jenkinsbro and our k8s component contains all the required logic and config to setup it.

Step III: create nexus docker registry

We will need a private repository to store our built docker images

Go to repositories: http://localhost:4000/#admin/repository/repositories
Click Create repository button
Select "docker (hosted)" item
Set Name "docker-registry"
Set HTTP flag and enter port "5000"
Click Create repository button at the bottom

Step IV: create Jenkins job to handle petclinic changes

Go to Jenkins: http://localhost:8085/
Create new job "petclinic-build" type "Multibranch Pipeline"
- Add "Git" to the Branch Sources
- Set Project Repository "http://git-service/example-org/petclinic"
- Click Save button at the bottom - after that multibranch job will check the available branches and start the master build
Set the gitea webhook - it will trigger changes build using the git plugin:
- Go to gitea petclinic repo webhooks settings: http://localhost:3000/example-org/petclinic/settings/hooks
- Click Add Webhook, select "Gogs" item
- Set Payload URL to "http://jenkins-service/git/notifyCommit?url=http://git-service/example-org/petclinic"
- Click Add Webhook

Provide feedback

Saved searches

Use saved searches to filter your results more quickly