Is it Observable

Episode : k8S TroubleShooting Guide

This repository contains the files utilized during the tutorial presented in the dedicated IsItObservable episode related to the k8S troubleshooting guide.

this tutorial will also utilize the OpenTelemetry Operator with:

• the OpenTelemetry Demo
• the hipster-shop
• istio gitg
• All the observability data generated by the environment would be sent to Dynatrace.

Prerequisite

The following tools need to be install on your machine :

• jq
• kubectl
• git
• gcloud ( if you are using GKE)
• Helm

1.Create a Google Cloud Platform Project

PROJECT_ID="<your-project-id>"
gcloud services enable container.googleapis.com --project ${PROJECT_ID}
gcloud services enable monitoring.googleapis.com \
cloudtrace.googleapis.com \
clouddebugger.googleapis.com \
cloudprofiler.googleapis.com \
--project ${PROJECT_ID}

2.Create a GKE cluster

ZONE=europe-west3-a
NAME=isitobservable-troubleshooting
gcloud container clusters create ${NAME} --zone=${ZONE} --machine-type=e2-standard-4 --num-nodes=2

Getting started

Dynatrace Tenant

1. Dynatrace Tenant - start a trial

If you don't have any Dynatrace tenant , then I suggest to create a trial using the following link : Dynatrace Trial Once you have your Tenant save the Dynatrace tenant url in the variable DT_TENANT_URL (for example : https://dedededfrf.live.dynatrace.com)

DT_TENANT_URL=<YOUR TENANT Host>

2. Create the Dynatrace API Tokens

The dynatrace operator will require to have several tokens:

• Token to deploy and configure the various components
• Token to ingest metrics and Traces

Operator Token

One for the operator having the following scope:

• Create ActiveGate tokens
• Read entities
• Read Settings
• Write Settings
• Access problem and event feed, metrics and topology
• Read configuration
• Write configuration
• Paas integration - installer downloader

Save the value of the token . We will use it later to store in a k8S secret

API_TOKEN=<YOUR TOKEN VALUE>

Ingest data token

Create a Dynatrace token with the following scope:

• Ingest metrics (metrics.ingest)
• Ingest logs (logs.ingest)
• Ingest events (events.ingest)
• Ingest OpenTelemetry
• Read metrics

Save the value of the token . We will use it later to store in a k8S secret

DATA_INGEST_TOKEN=<YOUR TOKEN VALUE>

Istio

1. Download Istioctl

curl -L https://istio.io/downloadIstio | sh -

This command download the latest version of istio compatible with our operating system. 2. Add istioctl to you PATH

cd istio-1.22.0

this directory contains samples with addons . We will refer to it later.

export PATH=$PWD/bin:$PATH

4.Clone Github repo

git clone https://github.com/isitobservable/K8STroubleShoting
cd K8STroubleShoting

5. Deploy most of the components for The collector

The application will deploy the entire environment:

chmod 777 deployment.sh
./deployment.sh  --clustername "${NAME}" --dturl "${DT_TENANT_URL}" --dtingesttoken "${DATA_INGEST_TOKEN}" --dtoperatortoken "${API_TOKEN}" 

6. The api server dashboard

in the Dynatrace folder , you will find the json file containing the dashboard for the API server. To deploy it click , on dashboard

And click on upload , to upload `dynatrace/Api server health.json`

7. The kublet/kubeproxy dashboard

Similar to the previous step, upload dynatrace/kubelet & Kubeproxy.json

8. Troubleshooting workflow

you will find a workflow sending messages to the team with details on why their workload are failing. This Worflow would require to reconfigure the slack integraiton and to install the Slack application from the Hub.

To import the workflow you will need to click on worflow .

Click on Upload to import the following file: ddynatrace/wftpl_troubeshooting_k8s_workload.yaml