Use Microclimate to run an End-to-End DevOps environment on IBM Cloud Private

Table of Contents

• Introduction
• Requirements
• Getting Started
• Install Bluecompute Reference Architecture Application
• Import this project into Microclimate
• Exploring Microclimate Workspace
  • Edit code Section
  • Open app Section
  • App logs Section
  • App monitor Section
    • Optional: Trigger a Stress Load
  • Pipeline Section
• Create and Run an Automated Jenkins Pipelines
  • Create Jenkins Pipeline
  • View Jenkins Pipeline
  • Create GitHub Web Hook
  • Trigger the pipeline through a Git Commit Push
  • Open Jenkins and see the pipeline in action
  • Open the new Jenkins deployment
• Conclusion

Introduction

This project is part of the 'IBM Cloud Native Reference Architecture' suite, available at https://github.com/ibm-cloud-architecture/refarch-cloudnative-kubernetes

Note: Microclimate v18.03 was used at the time of writing.

The purpose of this project is to demonstrate an End-to-End DevOps experience on IBM Cloud Private(ICP). This project will show you how to create, deploy, and monitor an application via Microclimate on an ICP Environment. It will also show you how to setup an automated Jenkins Pipeline that's triggered by GitHub web hooks.

This is a simple Node.js application that has 2 parts:

• An HTML home page, located in public/index.html
• A backend API with endpoints at /, /health, and /catalog
  • The /catalog endpoint is a custom endpoint that will display data that comes from a separate application.

The project structure (generated by Microclimate template) is as follows:

• .bluemix: contains the required files to setup an Open Toolchain on IBM Cloud.
• chart: contains a Helm chart, which is used to deploy the app into ICP.
• load-test: contains the files to run a JMeter load test.
• public: contains the frontend's HTML files.
• server: contains the Node.JS route filess.
• static: contains the static images used on this README.
• root: The files here that concern us the most are:
  • Dockerfile: used to build the Docker image.
  • Jenkinsfile: used by Jenkins to run the pipelines.
  • package.json: defines the Node.js project, its dependencies, and the start command.

Requirements

• Fork this repository

  • This is needed in order to setup automated Jenkins pipelines.

• GitHub Personal Access Token

  • This token is needed to import the project into microclimate.
  • It will also be used to create automated Jenkins pipelines.

• IBM Cloud Private Cluster

  • Create a Kubernetes cluster in an on-premise datacenter. The community edition (IBM Cloud private-ce) is free of charge.
  • Follow the instructions here to install IBM Cloud private-ce.

• Helm (Kubernetes package manager)

  • Follow the instructions here to install it on your platform.
  • If using IBM Cloud Private version 2.1.0.2 or newer, we recommend you follow these instructions to install helm.

• Microclimate

  • Download Microclimate by opening this link in your browser and press the Download button.
  • Once your ICP cluster is ready open this link and follow the instructions in the Create a docker-registry secret and Patch the docker-registry secret into your ICP service account sections to setup your ICP cluster for microclimate.
  • Once you are done with the above, use the following command to install the microclimate chart:

  # At the time of this writing, 18.03 is the latest version
  $ cd /PATH/TO/MICROCLIMATE/DOWNLOAD/microclimate-18.03
  
  # Install microclimate chart
  $ helm install --name microclimate --set persistence.enabled=false \
      --set jenkins.Persistence.Enabled=false \
      --set jenkins.Master.ServiceType=NodePort \
      --set jenkins.Master.NodePort=30127 \
      stable/ibm-microclimate --tls

  Once deployed, the above command will give you instructions on how to access microclimate web ui. Please note that it will take 5-10 minutes for microclimate to be fully initialized.

  Also, the above command will install microclimate without persistence, which is fine for quick testing. However, we recommend to use persistence in order to retain microclimate and Jenkins data. To configure persistence on microclimate, please checkout the Provide persistent storage volumes section on microclimate installation page.

  NOTE: If you are using a version of ICP older than 2.1.0.2, you don't need to add the --tls at the end of the helm command.

Getting Started

For this demo, we will be doing the following things:

• Install bluecompute reference architecture application:
• Create a microclimate application by importing this repository
• Setup automated Jenkins CICD pipelines
• Trigger Jenkins pipelines

Install Bluecompute Reference Architecture Application

The microclimate application requires that the Bluecompute Microservices Reference Architecture Chart is installed as this application pulls data from it's catalog microservice to showcase how it can communicate with existing Kubernetes applications.

To deploy the Bluecompute Chart, use these instructions. Please note that it takes Bluecompute about 5-10 minutes to be fully ready after deployment.

To learn more about Bluecompute Microservices Reference Architecture, please checkout its repo here.

Import this project into Microclimate

Assuming you already forked this repository this project and installed microclimate, let's go ahead and open microclimate in your browser. If you lost the microclimate access instructions from the helm install command, you can get them back using this command:

$ helm status microclimate --tls

Once you open microclimate in the browser, click Accept to accept the Microclimate license agreement.

Now you click the Import project button.

This will open the following view, where you have enter the repository details and finalize the import.

After a few seconds, you will get the following view confirming the import. Now click the Edit code button, which takes you to the microclimate workspace.

You have successfully imported this project into Microclimate.

Exploring Microclimate Workspace

You will now be presented to the Microclimate workspace view for the bluecatalog project. In here you have access to 5 views, where Edit code is presented by default. Before explaining each view separately, let's first go over what happened after you imported the project into Microclimate:

• A new docker image was built & pushed to ICP's internal Docker registry.
  • You can see the bluecatalog image if you open your browser and go to https://ICP_IP:8443/console/images.
  • Alternatively, you can open your terminal and enter $ kubectl get images to see the bluecatalog image.
• A helm chart was deployed using the freshly pushed docker image.
  • You can see the bluecatalog-deployment by opening a browser and going to https://ICP_IP:8443/console/workloads/deployments, then search for bluecatalog-deployment.
  • Alternatively, you can open your terminal and enter the kubectl get deployments bluecatalog-deployment command to see the deployment.
  • NOTE: Since microclimate uses its own helm client, you won't be able to see the helm release in https://ICP_IP:8443/catalog/instances

Now that you know what happened in the background, let's explore microclimate project views a bit more.

Edit code Section

This section presents you with the online code editor, which has a built-in git commands and a CLI where you can push/pull commits from remote repositories. By editing and saving changes from this view, you are triggering a new docker image build & push and an update to the existing helm chart deployment.

Open app Section

This section (shown below) presents you with a view that lets you see what your app looks like, which is very useful for debugging. There is an Application URL that let's you enter different paths and queries. There is also an Open in new tab button that lets you open you app in a separate tab away from microclimate.

Since we are here, let's test adding the /catalog path to the Application URL and pressing Reload to see if it displays the data that it gathers from the Bluecompute Application, which demonstrates a successful connection.

If you see an output similar to the above, then that means that the bluecatalog microclimate app was successfully deployed and connected to the Bluecompute Application!

App logs Section

This section presents you with a view of the logs produced by the application, which is great for debugging.

Notice the logs we see above is generated by the /catalog request.

App monitor Section

This section presents you with 3 tabs.

• The Dashboard tab is a collection of dashboards for things like CPU, HTTP Incoming Requests, Memory, etc.
• The Profiling tab displays the data as Flame Graphs that you can click on to see the call stacks in the graph nodes.
• The Summary tab gives you a more concise summary of all the data presented in the other tabs, such as HTTP Requests (endpoints and corresponding # of hits and response times), environment, and resource usage.

Notice in the screenshot below the values generated by our requests in the CPU, HTTP Incoming Requests, and Memory graphs.

For more information on the Application Metrics for Node.js, check out this link.

Optional: Trigger a Stress Load

To trigger a stress load on the app, click the Run load button above and see how the dashboard metrics change with the new load.

Notice how the graph values change once the workloads are triggered. The values go up as the load increases, then go down as either the app crashes or the load decreases.

Pipeline Section

Last but not least, there is the Pipeline section, in which you can setup Jenkins pipelines that can be configured to automatically be triggered by git commit.

Click the Create pipeline button and let's examine how to create an automated Jenkins pipelines in the next section.

Create and Run an Automated Jenkins Pipelines

The Microclimate workspace is great for developing tweaks in your application that you can quickly see in the same workspace. However, if you want a separate environment that other colleagues (i.e. QA) can see and test after pushing your updates upstream, it helps to have an automated CICD pipeline that does all of that for you.

Luckily, Microclimate comes with a Jenkins instance that it can setup for you with minor configuration.

Create Jenkins Pipeline

On the previous section you were asked to click the Create pipeline button, which should present a view as follows:

Follow the instructions above to setup your pipeline, then click the Create pipeline button. If successful, you will get a confirmation view as follows:

View Jenkins Pipeline

To view your Jenkins pipeline on Jenkins, click the Open pipeline button shown in the above image, which will open Jenkins in a new tab. If for some reason, your browser gives you an error or an incomplete URL, try using http://ICP_IP:30127/job/default/job/bluecatalog/.

If you are asked to login, use the credentials below:

• Username: admin
• Password: admin

Note: We recommend that you change the above password to something more secure.

After successful login, you should see a browser window similar to the following:

Create GitHub Web Hook

In order for the Jenkins pipeline to trigger automatically when you push a git commit, you need to setup a GitHub webhook that notifies Jenkins to start the pipeline.

To create the web hook in GitHub, do the following:

1. From your repository in GitHub, click Settings, Webhooks, and then click Add webhook.
2. Enter the Payload URL: JENKINS_URL/git/notifyCommit?url=GITHUB_URL, replacing JENKINS_URL with the base URL of your Jenkins deployment (In this case, it is http://ICP_IP:30127) and GITHUB_URL with the URL of the GitHub repository. For example, http://ICP_IP:30127/git/notifyCommit?url=https://github.com/microclimate-demo/node.git.
3. Select the Let me select individual events option.
4. From the list of events that are displayed, select Pull requests and Pushes.
5. Click Add webhook.

Now your Jenkins pipeline should automatically run every time you do a git push with new commits.

Trigger the pipeline through a Git Commit Push

Now it's time to trigger the pipeline with a new commit and test whether the GitHub web hook in your fork works. To do that, let's do the following:

• Go to the built-in editor and open the public/index.html file.
• Edit the text in line 10 from Hello world! This is a StarterKit! to Hello world! This is a new change! or whatever you like.
• Save the file.
• Open a terminal window.
• Change to the project directory.

$ cd bluecatalog

NOTE: It is called bluecatalog instead of refarch-cloudnative-bluecompute-microclimate because microclimate names the project folder after the name field in package.json. If you cloned the repo in your machine, the project folder would be refarch-cloudnative-bluecompute-microclimate.

• Remove the current remote repository and add a new one.

$ git remote rm origin
$ git remote add origin https://github.com/YOUR_GITHUB_USERNAME/refarch-cloudnative-bluecompute-microclimate.git

• Commit & Push the change to the remote repository.

$ git add public/index.html
$ git commit -m "Changed the home page"
$ git push origin master

Note: You can perform the same steps above on your local machine by cloning your fork repo. When you do this, there is no need to add an origin remote url.

Open Jenkins and see the pipeline in action

Now that you pushed a commit to the remote repository and assuming that the webhooks were setup properly, the Jenkins pipeline should be triggered and you can see it running. To view the running pipeline and its console output, open the URL http://ICP_IP:30127/job/default/job/bluecatalog/job/master in your browser and do the following.

Once in the console output, you should see output for the following:

• Git checkout of remote URL.
• Docker image build and image push to ICP Docker registry.
• Setup of Microclimate Helm client.
• Deployment/Upgrade of Helm chart.
• You should see a Finished: SUCCESS text at the end of the output if everything was successful.

Note: The Jenkins pipelines use the microclimate-pipeline-deployments Kubernetes namespace to deploy and update the charts. This is separate from the microclimate deployments that occur by saving a file in the built-in code editor, which in our case would deploy the chart in the default namespace. Also, since the Jenkins pipelines use their own Helm client, the resulting Helm releases are not visible inside the ICP GUI, though the actual Kubernetes resources are.

For example, if you open your browser and go to https://ICP_IP:8443/console/workloads/deployments, you would be able to see both the microclimate deployment and the Jenkins pipeline deployment in the default and microclimate-pipeline-deployments namespaces, respectively. Here is what that would look like:

Open the new Jenkins deployment

To open the new Jenkins pipeline deployment in your browser. Open the browser to https://ICP_IP:8443/console/access/services and perform the following steps:

To open the application URL in a new browser tab, perform the following steps:

Now a new browser tab will open greeting you with the new message:

You have successfully created and ran a Jenkins pipeline that created a new Helm release with your new changes!

Conclusion

Congratulations on making it to the end! Let's recap on what you just did.

• You imported your fork of this project into Microclimate, which by itself does the following:
  • Build and push the project's Docker image into ICP's Docker Registry.
  • Updated the project's helm chart with the new Docker image and deployed it into default namespace.
  • Starts collecting the app logs.
  • Starts collecting Application Metrics.
• You explored Microclimate workspace built-in features, which include:
  • Code editor and CLI.
  • Test browser to view your application UI.
  • App logs.
  • Application Metrics.
  • Jenkins Pipelines.
• You tested the application endpoints and its connection with the Bluecompute Reference Architecture app.
• You ran a JMeter load from the Application Metrics GUI and saw the metrics value change.
• You created a Jenkins pipeline and setup GitHub triggers to automatically trigger the pipeline when you push new commits to the repository.
• You used the built-in code editor to make changes and push them to the remote repository, which triggered the pipeline.
• You opened Jenkins to see the pipeline run and saw its console output.
• You used ICP GUI to see the resulting Kubernetes resources.
• Lastly, you used ICP GUI to open the newly deployed application in a new browser tab.

You now have the knowledge to use Microclimate to set up a fully automated DevOps experience on IBM Cloud Private. Now you just need to rinse and repeat for any other applications.