tempo-mixin: add panel for Envoy Proxy sidecar, update tanka example …

…and jsonnet dependencies (grafana#1137) * tempo-mixin: add panel for Envoy Proxy sidecar, update tanka example and jsonnet dependencies Changes: ### Tempo / Writes dashboard Add a panel with Envoy metrics. This panel is to be used when the gateway is run with an Envoy Proxy sidecar (for HTTP2 load balancing). The metrics from Envoy expose the gRPC status codes, which are essential pieces of information when debugging ingress issues. The gateway only reports HTTP statuses, which is deceptive since a 200 OK HTTP packet might contain a gRPC message with an error. Example: if a request is denied by the ingester because it hits the limit, the distributor will return a HTTP 200 OK message containg a gRPC message with status 9 FAILED PRECONDITION. I've also added a text panel listing the gRPC status codes since these aren't common knowledge. It links to this doc: https://github.com/grpc/grpc/blob/master/doc/statuscodes.md ### tanka example - update prometheus config to always add `cluster` label, the tempo-mixin dashboards depend on this label - enable search in Grafana and Tempo - use a relative link to tempo-mixin, this makes it easier to quickly test changes I've also run `jb update` in tempo-mixin and tk examples, hence the large amount of vendor file being changed. * Update CHANGELOG.md * Regenerate tempo-mixin/yamls
yvrhdn · Nov 26, 2021 · a1a95b3 · a1a95b3
1 parent ba93a40
commit a1a95b3
Show file tree

Hide file tree

Showing 51 changed files with 443 additions and 19,005 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,7 @@
 * [ENHANCEMENT] Allow query disablement in vulture [#1117](https://github.com/grafana/tempo/pull/1117) (@zalegrala)
 * [ENHANCEMENT] Improve memory efficiency of compaction and block cutting. [#1121](https://github.com/grafana/tempo/pull/1121) [#1130](https://github.com/grafana/tempo/pull/1130) (@joe-elliott)
 * [ENHANCEMENT] Include metrics for configured limit overrides and defaults: tempo_limits_overrides, tempo_limits_defaults [#1089](https://github.com/grafana/tempo/pull/1089) (@zalegrala)
+* [ENHANCEMENT] Add Envoy Proxy panel to `Tempo / Writes` dashboard [#1137](https://github.com/grafana/tempo/pull/1137) (@kvrhdn) 
 * [BUGFIX] Fix defaults for MaxBytesPerTrace (ingester.max-bytes-per-trace) and MaxSearchBytesPerTrace (ingester.max-search-bytes-per-trace) (@bitprocessor)
 * [BUGFIX] Ignore empty objects during compaction [#1113](https://github.com/grafana/tempo/pull/1113) (@mdisibio)
 * [BUGFIX] Add process name to vulture traces to work around display issues [#1127](https://github.com/grafana/tempo/pull/1127) (@mdisibio)

diff --git a/example/tk/jsonnetfile.json b/example/tk/jsonnetfile.json
@@ -49,20 +49,20 @@
     {
       "source": {
         "git": {
-          "remote": "https://github.com/grafana/tempo.git",
-          "subdir": "operations/tempo-mixin"
+          "remote": "https://github.com/jsonnet-libs/k8s-libsonnet.git",
+          "subdir": "1.21"
         }
       },
       "version": "main"
     },
     {
       "source": {
         "git": {
-          "remote": "https://github.com/jsonnet-libs/k8s-libsonnet.git",
-          "subdir": "1.21"
+          "remote": "https://github.com/grafana/jsonnet-libs.git",
+          "subdir": "grafana-builder"
         }
       },
-      "version": "main"
+      "version": "master"
     }
   ],
   "legacyImports": true

diff --git a/example/tk/jsonnetfile.lock.json b/example/tk/jsonnetfile.lock.json
@@ -8,8 +8,8 @@
           "subdir": "grafana"
         }
       },
-      "version": "17eca514f990530c411c2e9411af5213dd4bd224",
-      "sum": "eqpZVw4x6nN8duDjOTvwVj6WkfZXYR+gUJnrpMhzSUY="
+      "version": "c4975f7c4a7ab4c21020c4afbf247aa49142174d",
+      "sum": "BHvjNVA6foCySv2Rz6EY+miuwdGnsQxkqaOsZJYAi30="
     },
     {
       "source": {
@@ -18,8 +18,8 @@
           "subdir": "grafana-builder"
         }
       },
-      "version": "71640f24289af8deca742763feec3c0dbd22b7ae",
-      "sum": "U34Nd1ViO2LZ3D8IzygPPRfUcy6zOgCnTMVHZ+9O/QE="
+      "version": "c4975f7c4a7ab4c21020c4afbf247aa49142174d",
+      "sum": "y8uA/daOROErelzoo2p1rtqABhUPArg2alsfcb0PQBk="
     },
     {
       "source": {
@@ -28,7 +28,7 @@
           "subdir": "ksonnet-util"
         }
       },
-      "version": "17eca514f990530c411c2e9411af5213dd4bd224",
+      "version": "c4975f7c4a7ab4c21020c4afbf247aa49142174d",
       "sum": "fFVlCoa/N0qiqTbDhZAEdRm2Vv76Z9Clxp3/haJ+PyA="
     },
     {
@@ -38,7 +38,7 @@
           "subdir": "kube-state-metrics/"
         }
       },
-      "version": "71640f24289af8deca742763feec3c0dbd22b7ae",
+      "version": "c4975f7c4a7ab4c21020c4afbf247aa49142174d",
       "sum": "kH7gD2rdqRtBujmCObN0ifNF/BkSZU8pleFRI8itkqY="
     },
     {
@@ -48,38 +48,18 @@
           "subdir": "memcached"
         }
       },
-      "version": "17eca514f990530c411c2e9411af5213dd4bd224",
+      "version": "c4975f7c4a7ab4c21020c4afbf247aa49142174d",
       "sum": "dTOeEux3t9bYSqP2L/uCuLo/wUDpCKH4w+4OD9fePUk="
     },
-    {
-      "source": {
-        "git": {
-          "remote": "https://github.com/grafana/jsonnet-libs.git",
-          "subdir": "mixin-utils"
-        }
-      },
-      "version": "71640f24289af8deca742763feec3c0dbd22b7ae",
-      "sum": "v6fuqqQp9rHZbsxN9o79QzOpUlwYZEJ84DxTCZMCYeU="
-    },
     {
       "source": {
         "git": {
           "remote": "https://github.com/grafana/jsonnet-libs.git",
           "subdir": "prometheus"
         }
       },
-      "version": "71640f24289af8deca742763feec3c0dbd22b7ae",
-      "sum": "/UcvaaBqbWfbPfpwX+IU00PwbTIy6hTBq9Af5C55Zpg="
-    },
-    {
-      "source": {
-        "git": {
-          "remote": "https://github.com/grafana/tempo.git",
-          "subdir": "operations/tempo-mixin"
-        }
-      },
-      "version": "5124375ebfb128a9e698c81934a4d81b2887f299",
-      "sum": "p2wS372OU1fuxn8t/DKLj6v1VAGeYd9EOz0MYNHdUf0="
+      "version": "c4975f7c4a7ab4c21020c4afbf247aa49142174d",
+      "sum": "jecVpzopHoNjOW5CjsUnMfYNQNtKGlvzONYtQQbkDqg="
     },
     {
       "source": {
@@ -88,8 +68,8 @@
           "subdir": "1.21"
         }
       },
-      "version": "f47562c4e3bfaf46a5a5bc6a9e1e76a4fc76c18e",
-      "sum": "R4vw3zTrNH2hlwtMliAcnZ6ZlY21ACZHZ2tTkeU7QbM="
+      "version": "efb8cd3f9e2a3562c24e10404d5ed8b6c2d67174",
+      "sum": "EZpcgrWS8k6QCOMYW7t4wnLpCRlaikXLMDKJNV9ZD5Q="
     }
   ],
   "legacyImports": false

diff --git a/example/tk/lib/dashboards/datasources.libsonnet b/example/tk/lib/dashboards/datasources.libsonnet
@@ -6,13 +6,12 @@ local grafana = import 'grafana/grafana.libsonnet';
       'Tempo',
       url,
       'tempo',
+      default=true,
     ),
   prometheus:
     grafana.datasource.new(
       'Prometheus',
       'http://prometheus/prometheus',
       'prometheus',
-      true,
-    ) +
-    grafana.datasource.withHttpMethod('POST'),
+    ),
 }
diff --git a/example/tk/lib/dashboards/grafana.libsonnet b/example/tk/lib/dashboards/grafana.libsonnet
@@ -7,11 +7,19 @@ local mixins = import 'mixins.libsonnet';
   deploy(frontend_url='http://query-frontend'):
     grafana
     + grafana.withReplicas(1)
-    + grafana.withImage('grafana/grafana:8.2.1')
+    + grafana.withImage('grafana/grafana:8.3.0-beta2')
     + grafana.withRootUrl('http://grafana')
     + grafana.withTheme('dark')
     + grafana.withAnonymous()
 
+    + grafana.withGrafanaIniConfig({
+      sections+: {
+        feature_toggles: {
+          enable: 'tempoSearch',
+        },
+      },
+    })
+
     + grafana.addDatasource('Tempo', datasources.tempo(frontend_url))
     + grafana.addDatasource('Prometheus', datasources.prometheus)
 

diff --git a/example/tk/lib/dashboards/mixins.libsonnet b/example/tk/lib/dashboards/mixins.libsonnet
@@ -1,4 +1,4 @@
-local tempo = import 'tempo-mixin/mixin.libsonnet';
+local tempo = import '../../../../operations/tempo-mixin/mixin.libsonnet';
 
 {
   tempo:

diff --git a/example/tk/lib/k.libsonnet b/example/tk/lib/k.libsonnet
@@ -1 +1 @@
-import 'github.com/jsonnet-libs/k8s-libsonnet/1.21/main.libsonnet'
+import '1.21/main.libsonnet'
diff --git a/example/tk/lib/metrics/prometheus.libsonnet b/example/tk/lib/metrics/prometheus.libsonnet
@@ -1,21 +1,30 @@
-local ksm = import 'github.com/grafana/jsonnet-libs/kube-state-metrics/main.libsonnet';
+local ksm = import 'kube-state-metrics/main.libsonnet';
 local prometheus = import 'prometheus/prometheus.libsonnet';
 local scrape_configs = import 'prometheus/scrape_configs.libsonnet';
 
-local namespace = 'default';
-
 prometheus {
-  ksm: ksm.new(namespace),
+  ksm: ksm.new($._config.namespace),
 
   _config+:: {
     prometheus_external_hostname: 'http://prometheus',
   },
 
   prometheus_config+:: {
     scrape_configs: [
-      scrape_configs.kubernetes_pods,
+      scrape_configs.kubernetes_pods + {
+        // add the label 'cluster' to every scraped metric
+        relabel_configs+: [
+          {
+            source_labels: ['__address__'],  // always exists
+            regex: '.*',  // always matches
+            target_label: 'cluster',
+            replacement: $._config.cluster,
+            action: 'replace',
+          },
+        ],
+      },
       scrape_configs.kube_dns,
-      ksm.scrape_config(namespace),
+      ksm.scrape_config($._config.namespace),
     ],
   },
 }
diff --git a/example/tk/readme.md b/example/tk/readme.md
@@ -45,22 +45,17 @@ store them in an S3 or GCS bucket.  See configuration docs or some of the other
 tk apply tempo-single-binary
 ```
 
-### View a trace
-After the applications are running check the load generators logs
+### Search traces
 
-```console
-# you can find the exact pod name using `kubectl get pods`
-kubectl logs synthetic-load-generator-???
-```
-```
-20/03/03 21:30:01 INFO ScheduledTraceGenerator: Emitted traceId e9f4add3ac7c7115 for service frontend route /product
-20/03/03 21:30:01 INFO ScheduledTraceGenerator: Emitted traceId 3890ea9c4d7fab00 for service frontend route /cart
-20/03/03 21:30:01 INFO ScheduledTraceGenerator: Emitted traceId c36fc5169bf0693d for service frontend route /cart
-20/03/03 21:30:01 INFO ScheduledTraceGenerator: Emitted traceId ebaf7d02b96b30fc for service frontend route /cart
-20/03/03 21:30:02 INFO ScheduledTraceGenerator: Emitted traceId 23a09a0efd0d1ef0 for service frontend route /cart
-```
+Once all pods are up and running you can search for traces in Grafana.
+Navigate to http://localhost:3000/explore and select the search tab.
+
+### Dashboards
+
+Dashboards from [tempo-mixin](../../operations/tempo-mixin) are loaded as well.
+You can find them at http://localhost:3000/dashboards.
 
-Extract a trace id and view it in Grafana at http://localhost:3000/explore
+Note: these dashboards currently only work well with the microservices deployment.
 
 ### Clean up
 

diff --git a/example/tk/tempo-microservices/main.jsonnet b/example/tk/tempo-microservices/main.jsonnet
@@ -14,7 +14,9 @@ minio + metrics + load + tempo {
   },
 
   _config+:: {
+    cluster: 'k3d',
     namespace: 'default',
+    search_enabled: true,
     compactor+: {
     },
     querier+: {

diff --git a/example/tk/tempo-single-binary/main.jsonnet b/example/tk/tempo-single-binary/main.jsonnet
@@ -12,7 +12,9 @@ metrics + load + tempo {
   },
 
   _config+:: {
+    cluster: 'k3d',
     namespace: 'default',
+    search_enabled: true,
     pvc_size: '30Gi',
     pvc_storage_class: 'local-path',
     receivers: {

diff --git a/example/tk/vendor/github.com/grafana/jsonnet-libs/grafana-builder/grafana.libsonnet b/example/tk/vendor/github.com/grafana/jsonnet-libs/grafana-builder/grafana.libsonnet
diff --git a/example/tk/vendor/github.com/grafana/jsonnet-libs/grafana/configmaps.libsonnet b/example/tk/vendor/github.com/grafana/jsonnet-libs/grafana/configmaps.libsonnet
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		import 'github.com/jsonnet-libs/k8s-libsonnet/1.21/main.libsonnet'
		import '1.21/main.libsonnet'