Is it Observable

Episode : OpenTelemetry Collector Observability

This repository contains the files utilized during the tutorial presented in the dedicated IsItObservable episode related to the Observability of your OpenTelemetry Collectors. this tutorial will also utilize the OpenTelemetry Operator with:

• the OpenTelemetry Demo

• the hipster-shop

• nginx ingress controller

• All the observability data generated by the environment would be sent to Dynatrace.

Prerequisite

The following tools need to be install on your machine :

• jq
• kubectl
• git
• gcloud ( if you are using GKE)
• Helm

1.Create a Google Cloud Platform Project

PROJECT_ID="<your-project-id>"
gcloud services enable container.googleapis.com --project ${PROJECT_ID}
gcloud services enable monitoring.googleapis.com \
cloudtrace.googleapis.com \
clouddebugger.googleapis.com \
cloudprofiler.googleapis.com \
--project ${PROJECT_ID}

2.Create a GKE cluster

ZONE=europe-west3-a
NAME=isitobservable-collector-observability
gcloud container clusters create ${NAME} --zone=${ZONE} --machine-type=e2-standard-4 --num-nodes=2

3. Clone Github repo

git clone  https://github.com/isitobservable/collector-observability
cd collector-observability

Getting started

Dynatrace Tenant

1. Dynatrace Tenant - start a trial

If you don't have any Dynatrace tenant , then I suggest to create a trial using the following link : Dynatrace Trial Once you have your Tenant save the Dynatrace tenant url in the variable DT_TENANT_URL (for example : https://dedededfrf.live.dynatrace.com)

DT_TENANT_URL=<YOUR TENANT Host>

2. Create the Dynatrace API Tokens

The dynatrace operator will require to have several tokens:

• Token to deploy and configure the various components
• Token to ingest metrics and Traces

Operator Token

One for the operator having the following scope:

• Create ActiveGate tokens
• Read entities
• Read Settings
• Write Settings
• Access problem and event feed, metrics and topology
• Read configuration
• Write configuration
• Paas integration - installer downloader

Save the value of the token . We will use it later to store in a k8S secret

API_TOKEN=<YOUR TOKEN VALUE>

Ingest data token

Create a Dynatrace token with the following scope:

• Ingest metrics (metrics.ingest)
• Ingest logs (logs.ingest)
• Ingest events (events.ingest)
• Ingest OpenTelemetry
• Read metrics

Save the value of the token . We will use it later to store in a k8S secret

DATA_INGEST_TOKEN=<YOUR TOKEN VALUE>

Deploy most of the components

The application will deploy the entire environment:

chmod 777 deployment.sh
./deployment.sh  --clustername "${NAME}" --dturl "${DT_TENANT_URL}" --dtingesttoken "${DATA_INGEST_TOKEN}" --dtoperatortoken "${API_TOKEN}" 

Tutorial Steps

Collector Dashboard

Let's deploy the dashboard located : dynatrace/OpenTelemtry Collector.json

In dynatrace , Open The Dashboard application and click on upload

This dashboard will keep track on the health of your collector:

• ressource usage
• GC
• pipeline health

The Collector Traces

Now let's have a look at the traces produced by the collector

You can open your Dynatrace Tenant, and open The Services application , you should find a dedicated service for our OpenTelemetry Collector

And if you open this service you will find all the various traces produced by the collector:

You can select one of the distributes traces available in this screen you will see the trace produced by the exporter :