Is it Observable

Episode : Tetragon vs Falco vs KubeArmor

This repository contains all the files required to run the test comparing Falco , Tetragon , Kubearmor and Tracee

this repository will use solutions to measure the usage of those agents, we will rely on:

• the OpenTelemetry Demo

• ungard application

• Goat application to generate security violation

• All the observability data generated by the environment would be sent to Dynatrace.

Prerequisite

The following tools need to be install on your machine :

• jq
• kubectl
• git
• gcloud ( if you are using GKE)
• Helm

1.Create a Google Cloud Platform Project

PROJECT_ID="<your-project-id>"
gcloud services enable container.googleapis.com --project ${PROJECT_ID}
gcloud services enable monitoring.googleapis.com \
cloudtrace.googleapis.com \
clouddebugger.googleapis.com \
cloudprofiler.googleapis.com \
--project ${PROJECT_ID}

2.Create a GKE cluster

ZONE=europe-west3-a
NAME=isitobservable-securitybenchmark
gcloud container clusters create ${NAME} --zone=${ZONE} --machine-type=e2-standard-4 --num-nodes=2

3. Clone Github repo

git clone  https://github.com/isitobservable/runtimesecuritybenchmark
cd runtimesecuritybenchmark

Getting started

Dynatrace Tenant

1. Dynatrace Tenant - start a trial

If you don't have any Dynatrace tenant , then I suggest to create a trial using the following link : Dynatrace Trial Once you have your Tenant save the Dynatrace tenant url in the variable DT_TENANT_URL (for example : https://dedededfrf.live.dynatrace.com)

DT_TENANT_URL=<YOUR TENANT Host>

2. Create the Dynatrace API Tokens

The dynatrace operator will require to have several tokens:

• Token to deploy and configure the various components
• Token to ingest metrics and Traces

Operator Token

One for the operator having the following scope:

• Create ActiveGate tokens
• Read entities
• Read Settings
• Write Settings
• Access problem and event feed, metrics and topology
• Read configuration
• Write configuration
• Paas integration - installer downloader

Save the value of the token . We will use it later to store in a k8S secret

API_TOKEN=<YOUR TOKEN VALUE>

Ingest data token

Create a Dynatrace token with the following scope:

• Ingest metrics (metrics.ingest)
• Ingest logs (logs.ingest)
• Ingest events (events.ingest)
• Ingest OpenTelemetry
• Read metrics

Save the value of the token . We will use it later to store in a k8S secret

DATA_INGEST_TOKEN=<YOUR TOKEN VALUE>

1. Deploy the environment without security solutions

The application will deploy the entire environment:

TYPE=nothing
chmod 777 deployment.sh
./deployment.sh  --clustername "${NAME}" --dturl "${DT_TENANT_URL}" --dtingesttoken "${DATA_INGEST_TOKEN}" --dtoperatortoken "${API_TOKEN}" --type "${TYPE}"

Wait 30min before launching the load test agains the applications:

kubectl apply -f opentelemetry/loadtest_job.yaml -n otel-demo

2. Deploy the environment with Falco

The application will deploy the entire environment:

kubectl delete -f opentelemetry/loadtest_job.yaml -n otel-demo
OLD=$TYPE
TYPE=falco
chmod 777 update.sh
./update.sh  --type "${TYPE}" --previous "${OLD}"

Wait 30min before launching the load test agains the applications:

kubectl apply -f opentelemetry/loadtest_job.yaml -n otel-demo

3. Deploy the environment with Tetragon

The application will deploy the entire environment:

kubectl delete -f opentelemetry/loadtest_job.yaml -n otel-demo
OLD=$TYPE
TYPE=tetragon
chmod 777 update.sh
./update.sh  --type "${TYPE}" --previous "${OLD}"

Wait 30min before launching the load test agains the applications:

kubectl apply -f opentelemetry/loadtest_job.yaml -n otel-demo

Let's measure without tracing policies

kubectl delete -f opentelemetry/loadtest_job.yaml -n otel-demo
kubectl delete -k tetragon
kubectl apply -f opentelemetry/loadtest_job.yaml -n otel-demo

4. Deploy the environment with Kubearmor

The application will deploy the entire environment:

kubectl delete -f opentelemetry/loadtest_job.yaml -n otel-demo
OLD=$TYPE
TYPE=kubearmor
chmod 777 update.sh
./update.sh  --type "${TYPE}" --previous "${OLD}"

Modifiy the relay server to send events and alerts in the logs of kubearmor

kubectl edit deployment kubearmor-relay -n kubearmor

ENABLE_STDOUT_LOGS, ENABLE_STDOUT_ALERTS and ENABLE_STDOUT_MSGS needs to be equal to true

Wait 30min before launching the load test agains the applications:

kubectl apply -f opentelemetry/loadtest_job.yaml -n otel-demo

Let's measure without tracing policies

kubectl delete -f opentelemetry/loadtest_job.yaml -n otel-demo
kubectl delete -k tetragon
kubectl apply -f opentelemetry/loadtest_job.yaml -n otel-demo

Let's measure with events produced in several namesapces

kubectl delete -f opentelemetry/loadtest_job.yaml -n otel-demo
kubectl annotate ns otel-demo kubearmor-visibility=network,file,network,capabilities --overwrite
kubectl annotate ns goat-app kubearmor-visibility=network,file,network,capabilities --overwrite
kubectl annotate ns unguard kubearmor-visibility=network,file,network,capabilities --overwrite
kubectl annotate ns default kubearmor-visibility=network,file,network,capabilities --overwrite
kubectl apply -f opentelemetry/loadtest_job.yaml -n otel-demo

Let's measure with events produced and Kubermor policies in several namesapces

kubectl delete -f opentelemetry/loadtest_job.yaml -n otel-demo
kubectl apply -k kubearmor/policies
kubectl apply -f opentelemetry/loadtest_job.yaml -n otel-demo

5. Deploy the environment with Tracee

The application will deploy the entire environment:

kubectl delete -f opentelemetry/loadtest_job.yaml -n otel-demo
OLD=$TYPE
TYPE=tracee
chmod 777 update.sh
./update.sh  --type "${TYPE}" --previous "${OLD}"

Wait 30min before launching the load test agains the applications:

kubectl apply -f opentelemetry/loadtest_job.yaml -n otel-demo

Issue in your k8S environment

if you are having issue with lots of workload in pending state. It is related to the unguard cronjob that create to many jobs after a while. TO resolve this you will need to run the following command:

kubectl delete -f unguard/cronjob.yaml -n unguard

once all the pods has been removed you can re-create the cronjob:

kubectl apply -f unguard/cronjob.yaml -n unguard