Is it Observable

Episode : KubeArmor

This repository contains the files utilized during the tutorial presented in the dedicated IsItObservable episode related to KubeArmor.

this tutorial will also utilize the OpenTelemetry Operator with:

• the OpenTelemetry Demo

• ungard application

• Kubearmor

• Goat application to generate security violation

• All the observability data generated by the environment would be sent to Dynatrace.

Prerequisite

The following tools need to be install on your machine :

• jq
• kubectl
• git
• gcloud ( if you are using GKE)
• Helm

1.Create a Google Cloud Platform Project

PROJECT_ID="<your-project-id>"
gcloud services enable container.googleapis.com --project ${PROJECT_ID}
gcloud services enable monitoring.googleapis.com \
cloudtrace.googleapis.com \
clouddebugger.googleapis.com \
cloudprofiler.googleapis.com \
--project ${PROJECT_ID}

2.Create a GKE cluster

ZONE=europe-west3-a
NAME=isitobservable-kubearmor
gcloud container clusters create ${NAME} --zone=${ZONE} --machine-type=e2-standard-4 --num-nodes=2

3. Clone Github repo

git clone  https://github.com/isitobservable/kubearmor
cd kubearmor

Getting started

Dynatrace Tenant

1. Dynatrace Tenant - start a trial

If you don't have any Dynatrace tenant , then I suggest to create a trial using the following link : Dynatrace Trial Once you have your Tenant save the Dynatrace tenant url in the variable DT_TENANT_URL (for example : https://dedededfrf.live.dynatrace.com)

DT_TENANT_URL=<YOUR TENANT Host>

2. Create the Dynatrace API Tokens

The dynatrace operator will require to have several tokens:

• Token to deploy and configure the various components
• Token to ingest metrics and Traces

Operator Token

One for the operator having the following scope:

• Create ActiveGate tokens
• Read entities
• Read Settings
• Write Settings
• Access problem and event feed, metrics and topology
• Read configuration
• Write configuration
• Paas integration - installer downloader

Save the value of the token . We will use it later to store in a k8S secret

API_TOKEN=<YOUR TOKEN VALUE>

Ingest data token

Create a Dynatrace token with the following scope:

• Ingest metrics (metrics.ingest)
• Ingest logs (logs.ingest)
• Ingest events (events.ingest)
• Ingest OpenTelemetry
• Read metrics

Save the value of the token . We will use it later to store in a k8S secret

DATA_INGEST_TOKEN=<YOUR TOKEN VALUE>

Istio

1. Download Istioctl

curl -L https://istio.io/downloadIstio | sh -

This command download the latest version of istio compatible with our operating system. 2. Add istioctl to you PATH

cd istio-1.23.1

this directory contains samples with addons . We will refer to it later.

export PATH=$PWD/bin:$PATH

Deploy most of the components

The application will deploy the entire environment:

chmod 777 deployment.sh
./deployment.sh  --clustername "${NAME}" --dturl "${DT_TENANT_URL}" --dtingesttoken "${DATA_INGEST_TOKEN}" --dtoperatortoken "${API_TOKEN}" 

Modifiy the relay server to send events and alerts in the logs of kubearmor

kubectl edit deployment kubearmor-relay -n kubearmor

ENABLE_STDOUT_LOGS, ENABLE_STDOUT_ALERTS and ENABLE_STDOUT_MSGS needs to be equal to true

Tutorial Steps

Kubearmor Notebook

Let's deploy the notebook located : dynatrace/notebook.json

In dynatrace , Open The Notebook application and click on upload

This notebook is an example on how we could take advantage of log processing with tetragon events.

KubeArmor Dashboard

Let's deploy the dashboard located : dynatrace/kubearmor.json

In dynatrace , Open The Dashboard application and click on upload

This dashboard will keep track on the health of tetragon:

• ressource usage
• the various rules
• The tetragon events

all those dashboard are uisng the logs to parse the kubearmor event

Using the Kubearmor collector receiver.

The value of using this receiver is that you don't have to turn on the loggin on the realy server. Modifiy the relay server to stop creating events and alerts in the logs of kubearmor

kubectl edit deployment kubearmor-relay -n kubearmor

ENABLE_STDOUT_LOGS, ENABLE_STDOUT_ALERTS and ENABLE_STDOUT_MSGS needs to be equal to false

Build the collector version

To build the collector i have created my own opentelemetry/collector/manifest.yaml that describe the plugins that i would like to add in the collector. I have also included the DOcker file that will build the right image : opentelemetry/Dockerfile

Deploy the new collector

kubectl apply -f opentelemetry/openTelemetry-manifest_statefulset_kubarmor.yaml

the value is that we don't have to parse the logs , an observability backedn will automatically manage each fields of the kubearmor event.

Limitation

The kubearmor receiver is currently only compatible with the version 0.96 of the collector. we are not able to take advantage of the latest updates done by the community.