SRE-Playground

This repository is a playground for SRE and reliable monitoring setup for demo purpose. E2E solution runs in local k8 clusters provisioned with kind.

Features

Kubernetes Cluster

• kind
  • with kube-proxy
    • CNI
      • kindnet
        • x86_64
        • arm64
      • cilium
        • x86_64
        • arm64
  • without kube-proxy
    • CNI
      • cilium
        • x86_64
        • arm64

Demo Application

• hipstershop (https://github.com/open-telemetry/opentelemetry-demo)
• simpleapp

Logging

• fluent-bit ➡️ Loki
  • with kube-proxy
  • without kube-proxy
• Loki

Traces

• OpenTelemetry ➡️ Jaeger
  • workload traces
  • kubernetes controleplane traces

Metrics

• Prometheus
  • kubernetes components metrics
  • workload metrics
  • node metrics
  • cilium metrics

Ingress

• nginx-ingress
  • with kube-proxy
  • without kube-proxy
• cilium
  • with kube-proxy
  • without kube-proxy

Loadbalancer

• Metallb

Testing

• load testing
  • hipster
• chaos testing
  • gremlin
  • chaos-mesh
    • hipster
      • network-faults
        • bandwidth trip
        • delay
        • partition
      • pod-faults
        • container kill
        • pod failure
        • pod kill
      • stress-conditions
        • cpu stress
        • memory stress

Golden signal correlation

Navigation from Logs to Traces

Tools & software requirement

1. Docker
2. Kind
3. kubectl
4. helm

Installation

1. Seting up DNS

There are two options here (Option 1.1 & 1.2). First using DNSMASQ for DNS resolution. It helps in avoiding host file entries update. Or you can use traditional ways of updating hostfile entries.

Option 1.1: Install and configure DNSMASQ

  # The script installs dnsmasq and registers a local dns server "sre-playground.devops.nakednerds.net"
  ./dns/setup.sh

Option 1.2: Add host file entries

command: sudo vi /etc/hosts
entries:
127.0.0.1 hipstershop.sre-playground.devops.nakednerds.net
127.0.0.1 grafana.sre-playground.devops.nakednerds.net
127.0.0.1 prometheus.sre-playground.devops.nakednerds.net
127.0.0.1 chaostest.sre-playground.devops.nakednerds.net
127.0.0.1 hubble.sre-playground.devops.nakednerds.net
127.0.0.1 jaeger.sre-playground.devops.nakednerds.net

2. Clone the repo in local

  git clone https://github.com/rkdutta/sre-playground.git

3. Execute the installtion script

with-kubeproxy and cilium cni (default)

./bootstrap.sh <cluster-name> <CNI> <w/o kube-proxy>

with-kubeproxy and kindnet cni

./bootstrap.sh sre-playground kindnet with-kubeproxy

without-kubeproxy and cilium cni

./bootstrap.sh sre-playground cilium without-kubeproxy

UI Access

hipster playground

hipstershop.sre-playground.devops.nakednerds.net

topology dashboard (hubble)

hubble.sre-playground.devops.nakednerds.net

grafana

grafana.sre-playground.devops.nakednerds.net

User ID: admin
Password: prom-operator

prometheus

prometheus.sre-playground.devops.nakednerds.net

loadtest

hipstershop.sre-playground.devops.nakednerds.net/loadgen/

tracing: jaeger

jaeger.sre-playground.devops.nakednerds.net/jaeger/

chaos-mesh

chaostest.sre-playground.devops.nakednerds.net

# Generate user access token
 kubectl -n platform create token chaos-dashboard

# Trigger testing 
kubectl apply -f chaos-mesh/workflows/chaos-workflow.yaml
kubectl apply -f chaos-mesh/experiments/pod-faults/container-kill.yaml

Reseting the cluster

./reset.sh
(Note: Control plane components and CNI will not be deleted)

Deleting the cluster

./teardown.sh