From e1d265f591bb4b6bfd4afdc54b255ca3f1929a5e Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Tue, 27 Feb 2024 14:44:52 -0700 Subject: [PATCH 1/8] Add Grafana dashboard and installation steps Problem: As a user, I want to know how to easily install prometheus and grafana to visualize my NGF metrics. Solution: Add basic installation steps for both prometheus and grafana, and provide a sample dashboard (based on the nginx-prometheus-exporter dashboard) --- site/content/how-to/monitoring/prometheus.md | 110 ++-- site/static/grafana-dashboard.json | 556 +++++++++++++++++++ 2 files changed, 632 insertions(+), 34 deletions(-) create mode 100644 site/static/grafana-dashboard.json diff --git a/site/content/how-to/monitoring/prometheus.md b/site/content/how-to/monitoring/prometheus.md index b4da07507b..a7d37de00c 100644 --- a/site/content/how-to/monitoring/prometheus.md +++ b/site/content/how-to/monitoring/prometheus.md @@ -1,6 +1,6 @@ --- -title: "Prometheus Metrics" -description: "Learn how to monitor your NGINX Gateway Fabric effectively. This guide provides easy steps for configuring and understanding key performance metrics using Prometheus." +title: "Prometheus Metrics and Grafana" +description: "Learn how to monitor your NGINX Gateway Fabric effectively. This guide provides easy steps for configuring and understanding key performance metrics using Prometheus and Grafana." weight: 100 toc: true docs: "DOCS-1418" @@ -11,13 +11,85 @@ docs: "DOCS-1418" ## Overview -NGINX Gateway Fabric metrics are displayed in [Prometheus](https://prometheus.io/) format, simplifying monitoring. You can track NGINX and controller-runtime metrics through a metrics server orchestrated by the controller-runtime package. These metrics are enabled by default and can be accessed on HTTP port `9113`. - +NGINX Gateway Fabric metrics are displayed in [Prometheus](https://prometheus.io/) format, simplifying monitoring. These metrics are served via a metrics server orchestrated by the controller-runtime package on HTTP port `9113`. When installed, Prometheus automatically scrapes this port and collects metrics. [Grafana](https://grafana.com/) can be used for rich visualization of these metrics. {{}} Metrics are served over HTTP by default. Enabling HTTPS will secure the metrics endpoint with a self-signed certificate. When using HTTPS, adjust the Prometheus Pod scrape settings by adding the `insecure_skip_verify` flag to handle the self-signed certificate. For further details, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config). {{}} +## Installing Prometheus and Grafana + +### Prometheus + +```shell +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +helm repo update +helm install prometheus prometheus-community/prometheus -n monitoring --create-namespace --set server.global.scrape_interval=15s +``` + +Once running, you can access the Prometheus dashboard by using port-forwarding: + +```shell +kubectl -n monitoring port-forward svc/prometheus-server 9090:80 +``` + +Visit [http://127.0.0.1:9090](http://127.0.0.1:9090) to view the dashboard. + +### Grafana + + +```shell +helm repo add grafana https://grafana.github.io/helm-charts +helm repo update +helm install grafana grafana/grafana -n monitoring --create-namespace +``` + +Once running, you can access the Grafana dashboard by using port-forwarding: + +```shell +kubectl -n monitoring port-forward svc/grafana 3000:80 +``` + +Visit [http://127.0.0.1:3000](http://127.0.0.1:3000) to view the Grafana UI. + +#### Configuring Grafana + +In the Grafana UI menu, go to `Connections` then `Data sources`. Add your Prometheus service (`http://prometheus-server.monitoring.svc`) as a data source. + +Download the following sample dashboard and Import as a new Dashboard in the Grafana UI. + +{{< fa "download" >}} {{< link "static/grafana-dashboard.json" "`ngf-grafana-dashboard.json`" >}} + +## Available metrics in NGINX Gateway Fabric + +NGINX Gateway Fabric provides a variety of metrics to assist in monitoring and analyzing performance. These metrics are categorized as follows: + +### NGINX/NGINX Plus metrics + +NGINX metrics, essential for monitoring specific NGINX operations, include details like the total number of accepted client connections. For a complete list of available NGINX/NGINX Plus metrics, refer to the [NGINX Prometheus Exporter developer docs](https://github.com/nginxinc/nginx-prometheus-exporter#exported-metrics). + +These metrics use the `nginx_gateway_fabric` namespace and include the `class` label, indicating the NGINX Gateway class. For example, `nginx_gateway_fabric_connections_accepted{class="nginx"}`. + +### NGINX Gateway Fabric metrics + +Metrics specific to the NGINX Gateway Fabric include: + +- `nginx_reloads_total`: Counts successful NGINX reloads. +- `nginx_reload_errors_total`: Counts NGINX reload failures. +- `nginx_stale_config`: Indicates if NGINX Gateway Fabric couldn't update NGINX with the latest configuration, resulting in a stale version. +- `nginx_last_reload_milliseconds`: Time in milliseconds for NGINX reloads. +- `event_batch_processing_milliseconds`: Time in milliseconds to process batches of Kubernetes events. + +All these metrics are under the `nginx_gateway_fabric` namespace and include a `class` label set to the Gateway class of NGINX Gateway Fabric. For example, `nginx_gateway_fabric_nginx_reloads_total{class="nginx"}`. + +### Controller-runtime metrics + +Provided by the [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) library, these metrics cover a range of aspects: + +- General resource usage like CPU and memory. +- Go runtime metrics such as the number of Go routines, garbage collection duration, and Go version. +- Controller-specific metrics, including reconciliation errors per controller, length of the reconcile queue, and reconciliation latency. + ## How to change the default metrics configuration Configuring NGINX Gateway Fabric for monitoring is straightforward. You can change metric settings using Helm or Kubernetes manifests, depending on your setup. @@ -85,33 +157,3 @@ For enhanced security with HTTPS: prometheus.io/scheme: "https" <...> ``` - -## Available metrics in NGINX Gateway Fabric - -NGINX Gateway Fabric provides a variety of metrics to assist in monitoring and analyzing performance. These metrics are categorized as follows: - -### NGINX/NGINX Plus metrics - -NGINX metrics, essential for monitoring specific NGINX operations, include details like the total number of accepted client connections. For a complete list of available NGINX/NGINX Plus metrics, refer to the [NGINX Prometheus Exporter developer docs](https://github.com/nginxinc/nginx-prometheus-exporter#exported-metrics). - -These metrics use the `nginx_gateway_fabric` namespace and include the `class` label, indicating the NGINX Gateway class. For example, `nginx_gateway_fabric_connections_accepted{class="nginx"}`. - -### NGINX Gateway Fabric metrics - -Metrics specific to the NGINX Gateway Fabric include: - -- `nginx_reloads_total`: Counts successful NGINX reloads. -- `nginx_reload_errors_total`: Counts NGINX reload failures. -- `nginx_stale_config`: Indicates if NGINX Gateway Fabric couldn't update NGINX with the latest configuration, resulting in a stale version. -- `nginx_last_reload_milliseconds`: Time in milliseconds for NGINX reloads. -- `event_batch_processing_milliseconds`: Time in milliseconds to process batches of Kubernetes events. - -All these metrics are under the `nginx_gateway_fabric` namespace and include a `class` label set to the Gateway class of NGINX Gateway Fabric. For example, `nginx_gateway_fabric_nginx_reloads_total{class="nginx"}`. - -### Controller-runtime metrics - -Provided by the [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) library, these metrics cover a range of aspects: - -- General resource usage like CPU and memory. -- Go runtime metrics such as the number of Go routines, garbage collection duration, and Go version. -- Controller-specific metrics, including reconciliation errors per controller, length of the reconcile queue, and reconciliation latency. diff --git a/site/static/grafana-dashboard.json b/site/static/grafana-dashboard.json new file mode 100644 index 0000000000..9c882ea8f4 --- /dev/null +++ b/site/static/grafana-dashboard.json @@ -0,0 +1,556 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "description": "Sample dashboard for NGINX Gateway Fabric", + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": 5, + "links": [], + "liveNow": false, + "panels": [ + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 5, + "panels": [], + "title": "Status", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [ + { + "options": { + "0": { + "index": 0, + "text": "Down" + }, + "1": { + "index": 1, + "text": "Up" + } + }, + "type": "value" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "semi-dark-red", + "value": null + }, + { + "color": "#EAB839", + "value": 1 + }, + { + "color": "semi-dark-green", + "value": 1 + } + ] + }, + "unit": "none", + "unitScale": true + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 0, + "y": 1 + }, + "id": 3, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "10.3.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "disableTextWrap": false, + "editorMode": "builder", + "expr": "nginx_gateway_fabric_up{instance=~\"$instance\"}", + "fullMetaSearch": false, + "includeNullMetadata": true, + "instant": false, + "legendFormat": "", + "range": true, + "refId": "A", + "useBackend": false + } + ], + "title": "NGINX Status for $instance", + "type": "stat" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 5 + }, + "id": 6, + "panels": [], + "title": "Metrics", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Connections (rate)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 1, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "reqps", + "unitScale": true + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 6 + }, + "id": 1, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "disableTextWrap": false, + "editorMode": "code", + "expr": "irate(nginx_gateway_fabric_connections_accepted{instance=~\"$instance\"}[5m])", + "fullMetaSearch": false, + "includeNullMetadata": false, + "instant": false, + "interval": "", + "legendFormat": "{{instance}} accepted", + "range": true, + "refId": "A", + "useBackend": false + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "irate(nginx_gateway_fabric_connections_handled{instance=~\"$instance\"}[5m])", + "hide": false, + "instant": false, + "legendFormat": "{{instance}} handled", + "range": true, + "refId": "B" + } + ], + "title": "Processed Connections", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Connections", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 1, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short", + "unitScale": true + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 6 + }, + "id": 4, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "nginx_gateway_fabric_connections_active{instance=~\"$instance\"}", + "instant": false, + "legendFormat": "{{instance}} active", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "nginx_gateway_fabric_connections_reading{instance=~\"$instance\"}", + "hide": false, + "instant": false, + "legendFormat": "{{instance}} reading", + "range": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "nginx_gateway_fabric_connections_waiting{instance=~\"$instance\"}", + "hide": false, + "instant": false, + "legendFormat": "{{instance}} waiting", + "range": true, + "refId": "C" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "nginx_gateway_fabric_connections_writing{instance=~\"$instance\"}", + "hide": false, + "instant": false, + "legendFormat": "{{instance}} writing", + "range": true, + "refId": "D" + } + ], + "title": "Active Connections", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 1, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "reqps", + "unitScale": true + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 16 + }, + "id": 2, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "disableTextWrap": false, + "editorMode": "builder", + "expr": "irate(nginx_gateway_fabric_http_requests_total{instance=~\"$instance\"}[5m])", + "fullMetaSearch": false, + "includeNullMetadata": false, + "instant": false, + "legendFormat": "{{instance}} total requests", + "range": true, + "refId": "A", + "useBackend": false + } + ], + "title": "Total Requests", + "type": "timeseries" + } + ], + "refresh": "5s", + "schemaVersion": 39, + "tags": [ + "nginx-gateway-fabric" + ], + "templating": { + "list": [ + { + "current": { + "selected": true, + "text": "default", + "value": "default" + }, + "hide": 0, + "includeAll": false, + "label": "datasource", + "multi": false, + "name": "DS_PROMETHEUS", + "options": [], + "query": "prometheus", + "queryValue": "", + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "type": "datasource" + }, + { + "current": { + "selected": true, + "text": [ + "All" + ], + "value": [ + "$__all" + ] + }, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "definition": "label_values(nginx_gateway_fabric_up,instance)", + "hide": 0, + "includeAll": true, + "multi": true, + "name": "instance", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(nginx_gateway_fabric_up,instance)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "sort": 0, + "type": "query" + } + ] + }, + "time": { + "from": "now-15m", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "NGINX Gateway Fabric", + "uid": "cdb1c6f6-7c77-4cee-a177-593f41364dbe", + "version": 1, + "weekStart": "" +} From 7de0679566538cb8a97f646da62df8562ccb8fda Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Wed, 28 Feb 2024 10:55:14 -0700 Subject: [PATCH 2/8] Code review --- site/content/how-to/monitoring/prometheus.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/site/content/how-to/monitoring/prometheus.md b/site/content/how-to/monitoring/prometheus.md index a7d37de00c..b1236fdf5b 100644 --- a/site/content/how-to/monitoring/prometheus.md +++ b/site/content/how-to/monitoring/prometheus.md @@ -27,10 +27,10 @@ helm repo update helm install prometheus prometheus-community/prometheus -n monitoring --create-namespace --set server.global.scrape_interval=15s ``` -Once running, you can access the Prometheus dashboard by using port-forwarding: +Once running, you can access the Prometheus dashboard by using port-forwarding in the background: ```shell -kubectl -n monitoring port-forward svc/prometheus-server 9090:80 +kubectl port-forward -n monitoring svc/prometheus-server 9090:80 & ``` Visit [http://127.0.0.1:9090](http://127.0.0.1:9090) to view the dashboard. @@ -44,21 +44,27 @@ helm repo update helm install grafana grafana/grafana -n monitoring --create-namespace ``` -Once running, you can access the Grafana dashboard by using port-forwarding: +Once running, you can access the Grafana dashboard by using port-forwarding in the background: ```shell -kubectl -n monitoring port-forward svc/grafana 3000:80 +kubectl port-forward -n monitoring svc/grafana 3000:80 & ``` Visit [http://127.0.0.1:3000](http://127.0.0.1:3000) to view the Grafana UI. +The username for login is `admin`. The password can be acquired by running: + +```shell +kubectl get secret -n monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo +``` + #### Configuring Grafana In the Grafana UI menu, go to `Connections` then `Data sources`. Add your Prometheus service (`http://prometheus-server.monitoring.svc`) as a data source. Download the following sample dashboard and Import as a new Dashboard in the Grafana UI. -{{< fa "download" >}} {{< link "static/grafana-dashboard.json" "`ngf-grafana-dashboard.json`" >}} +{{< fa "download" >}} {{< link "grafana-dashboard.json" "`ngf-grafana-dashboard.json`" >}} ## Available metrics in NGINX Gateway Fabric From eea541e0103ce6db33c2941fac45e70fd912a74f Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Thu, 29 Feb 2024 08:09:20 -0700 Subject: [PATCH 3/8] Update download shortcorde --- site/content/how-to/monitoring/prometheus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/how-to/monitoring/prometheus.md b/site/content/how-to/monitoring/prometheus.md index b1236fdf5b..a5c03efad5 100644 --- a/site/content/how-to/monitoring/prometheus.md +++ b/site/content/how-to/monitoring/prometheus.md @@ -64,7 +64,7 @@ In the Grafana UI menu, go to `Connections` then `Data sources`. Add your Promet Download the following sample dashboard and Import as a new Dashboard in the Grafana UI. -{{< fa "download" >}} {{< link "grafana-dashboard.json" "`ngf-grafana-dashboard.json`" >}} +{{< download "static/grafana-dashboard.json" "ngf-grafana-dashboard.json" >}} ## Available metrics in NGINX Gateway Fabric From 7888db61f7a8cacfc8528272b721280b24734cdf Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Thu, 29 Feb 2024 08:49:13 -0700 Subject: [PATCH 4/8] Code review --- site/content/how-to/monitoring/prometheus.md | 22 +- site/static/grafana-dashboard.json | 271 ++++++++++++++++++- 2 files changed, 277 insertions(+), 16 deletions(-) diff --git a/site/content/how-to/monitoring/prometheus.md b/site/content/how-to/monitoring/prometheus.md index a5c03efad5..70ff7bea8e 100644 --- a/site/content/how-to/monitoring/prometheus.md +++ b/site/content/how-to/monitoring/prometheus.md @@ -1,6 +1,6 @@ --- -title: "Prometheus Metrics and Grafana" -description: "Learn how to monitor your NGINX Gateway Fabric effectively. This guide provides easy steps for configuring and understanding key performance metrics using Prometheus and Grafana." +title: "Prometheus Metrics" +description: "This document describes how to monitor NGINX Gateway Fabric using Prometheus and Grafana. It explains installation and configuration, as well as what metrics are available." weight: 100 toc: true docs: "DOCS-1418" @@ -11,7 +11,7 @@ docs: "DOCS-1418" ## Overview -NGINX Gateway Fabric metrics are displayed in [Prometheus](https://prometheus.io/) format, simplifying monitoring. These metrics are served via a metrics server orchestrated by the controller-runtime package on HTTP port `9113`. When installed, Prometheus automatically scrapes this port and collects metrics. [Grafana](https://grafana.com/) can be used for rich visualization of these metrics. +NGINX Gateway Fabric metrics are displayed in [Prometheus](https://prometheus.io/) format. These metrics are served through a metrics server orchestrated by the controller-runtime package on HTTP port `9113`. When installed, Prometheus automatically scrapes this port and collects metrics. [Grafana](https://grafana.com/) can be used for rich visualization of these metrics. {{}} Metrics are served over HTTP by default. Enabling HTTPS will secure the metrics endpoint with a self-signed certificate. When using HTTPS, adjust the Prometheus Pod scrape settings by adding the `insecure_skip_verify` flag to handle the self-signed certificate. For further details, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config). @@ -19,6 +19,8 @@ Metrics are served over HTTP by default. Enabling HTTPS will secure the metrics ## Installing Prometheus and Grafana +{{< note >}}These installations are for demonstration purposes and have not been tuned for a production environment.{{< /note >}} + ### Prometheus ```shell @@ -64,21 +66,21 @@ In the Grafana UI menu, go to `Connections` then `Data sources`. Add your Promet Download the following sample dashboard and Import as a new Dashboard in the Grafana UI. -{{< download "static/grafana-dashboard.json" "ngf-grafana-dashboard.json" >}} + ## Available metrics in NGINX Gateway Fabric -NGINX Gateway Fabric provides a variety of metrics to assist in monitoring and analyzing performance. These metrics are categorized as follows: +NGINX Gateway Fabric provides a variety of metrics for monitoring and analyzing performance. These metrics are categorized as follows: ### NGINX/NGINX Plus metrics -NGINX metrics, essential for monitoring specific NGINX operations, include details like the total number of accepted client connections. For a complete list of available NGINX/NGINX Plus metrics, refer to the [NGINX Prometheus Exporter developer docs](https://github.com/nginxinc/nginx-prometheus-exporter#exported-metrics). +NGINX metrics cover specific NGINX operations such as the total number of accepted client connections. For a complete list of available NGINX/NGINX Plus metrics, refer to the [NGINX Prometheus Exporter developer docs](https://github.com/nginxinc/nginx-prometheus-exporter#exported-metrics). -These metrics use the `nginx_gateway_fabric` namespace and include the `class` label, indicating the NGINX Gateway class. For example, `nginx_gateway_fabric_connections_accepted{class="nginx"}`. +These metrics use the `nginx_gateway_fabric` namespace and include the `class` label, indicating the NGINX Gateway class. For example, `nginx_gateway_fabric_connections_accepted{class="nginx"}`. ### NGINX Gateway Fabric metrics -Metrics specific to the NGINX Gateway Fabric include: +Metrics specific to NGINX Gateway Fabric include: - `nginx_reloads_total`: Counts successful NGINX reloads. - `nginx_reload_errors_total`: Counts NGINX reload failures. @@ -90,7 +92,7 @@ All these metrics are under the `nginx_gateway_fabric` namespace and include a ` ### Controller-runtime metrics -Provided by the [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) library, these metrics cover a range of aspects: +Provided by the [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) library, these metrics include: - General resource usage like CPU and memory. - Go runtime metrics such as the number of Go routines, garbage collection duration, and Go version. @@ -98,7 +100,7 @@ Provided by the [controller-runtime](https://github.com/kubernetes-sigs/controll ## How to change the default metrics configuration -Configuring NGINX Gateway Fabric for monitoring is straightforward. You can change metric settings using Helm or Kubernetes manifests, depending on your setup. +You can configure monitoring metrics for NGINX Gateway Fabric using Helm or Manifests. ### Using Helm diff --git a/site/static/grafana-dashboard.json b/site/static/grafana-dashboard.json index 9c882ea8f4..10a4cd3c15 100644 --- a/site/static/grafana-dashboard.json +++ b/site/static/grafana-dashboard.json @@ -19,7 +19,7 @@ "editable": true, "fiscalYearStartMonth": 0, "graphTooltip": 0, - "id": 5, + "id": 1, "links": [], "liveNow": false, "panels": [ @@ -129,7 +129,7 @@ "type": "stat" }, { - "collapsed": true, + "collapsed": false, "gridPos": { "h": 1, "w": 24, @@ -226,7 +226,7 @@ }, "disableTextWrap": false, "editorMode": "code", - "expr": "irate(nginx_gateway_fabric_connections_accepted{instance=~\"$instance\"}[5m])", + "expr": "irate(nginx_gateway_fabric_connections_accepted{instance=~\"$instance\"}[1m])", "fullMetaSearch": false, "includeNullMetadata": false, "instant": false, @@ -242,7 +242,7 @@ "uid": "${DS_PROMETHEUS}" }, "editorMode": "code", - "expr": "irate(nginx_gateway_fabric_connections_handled{instance=~\"$instance\"}[5m])", + "expr": "irate(nginx_gateway_fabric_connections_handled{instance=~\"$instance\"}[1m])", "hide": false, "instant": false, "legendFormat": "{{instance}} handled", @@ -469,8 +469,8 @@ "uid": "${DS_PROMETHEUS}" }, "disableTextWrap": false, - "editorMode": "builder", - "expr": "irate(nginx_gateway_fabric_http_requests_total{instance=~\"$instance\"}[5m])", + "editorMode": "code", + "expr": "irate(nginx_gateway_fabric_http_requests_total{instance=~\"$instance\"}[1m])", "fullMetaSearch": false, "includeNullMetadata": false, "instant": false, @@ -482,6 +482,265 @@ ], "title": "Total Requests", "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 1, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unitScale": true + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 24 + }, + "id": 8, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "10.3.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + }, + "disableTextWrap": false, + "editorMode": "code", + "expr": "irate(nginx_gateway_fabric_nginx_reloads_total{instance=~\"$instance\"}[1m])", + "fullMetaSearch": false, + "includeNullMetadata": false, + "instant": false, + "legendFormat": "{{instance}}", + "range": true, + "refId": "A", + "useBackend": false + } + ], + "title": "Total NGINX Reloads Rate", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unitScale": true + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 6, + "x": 12, + "y": 24 + }, + "id": 9, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "10.3.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + }, + "disableTextWrap": false, + "editorMode": "builder", + "expr": "nginx_gateway_fabric_nginx_reload_errors_total{instance=~\"$instance\"}", + "fullMetaSearch": false, + "includeNullMetadata": true, + "instant": false, + "legendFormat": "{{instance}}", + "range": true, + "refId": "A", + "useBackend": false + } + ], + "title": "Total NGINX Reload Errors", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [ + { + "options": { + "0": { + "color": "semi-dark-green", + "index": 0, + "text": "Up to date" + }, + "1": { + "color": "semi-dark-red", + "index": 1, + "text": "Stale" + } + }, + "type": "value" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "semi-dark-red", + "value": 1 + } + ] + }, + "unitScale": true + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 6, + "x": 18, + "y": 24 + }, + "id": 10, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "10.3.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + }, + "disableTextWrap": false, + "editorMode": "builder", + "expr": "nginx_gateway_fabric_nginx_stale_config{instance=~\"$instance\"}", + "fullMetaSearch": false, + "includeNullMetadata": true, + "instant": false, + "legendFormat": "__auto", + "range": true, + "refId": "A", + "useBackend": false + } + ], + "title": "NGINX Config State", + "type": "stat" } ], "refresh": "5s", From 2b63134ea1baf465c7eb95394211004de021dc01 Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Thu, 29 Feb 2024 08:54:58 -0700 Subject: [PATCH 5/8] Add download link back in --- site/content/how-to/monitoring/prometheus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/how-to/monitoring/prometheus.md b/site/content/how-to/monitoring/prometheus.md index 70ff7bea8e..57227d5fb7 100644 --- a/site/content/how-to/monitoring/prometheus.md +++ b/site/content/how-to/monitoring/prometheus.md @@ -66,7 +66,7 @@ In the Grafana UI menu, go to `Connections` then `Data sources`. Add your Promet Download the following sample dashboard and Import as a new Dashboard in the Grafana UI. - +{{< download "static/grafana-dashboard.json" "ngf-grafana-dashboard.json" >}} ## Available metrics in NGINX Gateway Fabric From bd9f5a8576160b57245501ca503ac2e36eb79d8e Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Tue, 5 Mar 2024 08:11:42 -0700 Subject: [PATCH 6/8] Update dashboard --- site/static/grafana-dashboard.json | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/site/static/grafana-dashboard.json b/site/static/grafana-dashboard.json index 10a4cd3c15..0c3c40392d 100644 --- a/site/static/grafana-dashboard.json +++ b/site/static/grafana-dashboard.json @@ -486,7 +486,7 @@ { "datasource": { "type": "prometheus", - "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + "uid": "${DS_PROMETHEUS}" }, "fieldConfig": { "defaults": { @@ -563,7 +563,7 @@ { "datasource": { "type": "prometheus", - "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + "uid": "${DS_PROMETHEUS}" }, "disableTextWrap": false, "editorMode": "code", @@ -583,7 +583,7 @@ { "datasource": { "type": "prometheus", - "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + "uid": "${DS_PROMETHEUS}" }, "fieldConfig": { "defaults": { @@ -636,7 +636,7 @@ { "datasource": { "type": "prometheus", - "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + "uid": "${DS_PROMETHEUS}" }, "disableTextWrap": false, "editorMode": "builder", @@ -656,7 +656,7 @@ { "datasource": { "type": "prometheus", - "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + "uid": "${DS_PROMETHEUS}" }, "fieldConfig": { "defaults": { @@ -725,7 +725,7 @@ { "datasource": { "type": "prometheus", - "uid": "cb451057-d630-4956-9a0e-f19cdc9c8097" + "uid": "${DS_PROMETHEUS}" }, "disableTextWrap": false, "editorMode": "builder", @@ -752,7 +752,7 @@ "list": [ { "current": { - "selected": true, + "selected": false, "text": "default", "value": "default" }, From 79e4be3615c78a96d4a8c2b3ad80c917241fde1e Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Thu, 7 Mar 2024 08:11:53 -0700 Subject: [PATCH 7/8] remove static --- site/content/how-to/monitoring/prometheus.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/how-to/monitoring/prometheus.md b/site/content/how-to/monitoring/prometheus.md index 57227d5fb7..857c0c2bb6 100644 --- a/site/content/how-to/monitoring/prometheus.md +++ b/site/content/how-to/monitoring/prometheus.md @@ -66,7 +66,7 @@ In the Grafana UI menu, go to `Connections` then `Data sources`. Add your Promet Download the following sample dashboard and Import as a new Dashboard in the Grafana UI. -{{< download "static/grafana-dashboard.json" "ngf-grafana-dashboard.json" >}} +{{< download "grafana-dashboard.json" "ngf-grafana-dashboard.json" >}} ## Available metrics in NGINX Gateway Fabric From 095b8652a8113f7bd5b9602ca046e580b45cf457 Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Fri, 8 Mar 2024 09:23:57 -0700 Subject: [PATCH 8/8] update hugo --- site/go.mod | 2 +- site/go.sum | 12 ++---------- 2 files changed, 3 insertions(+), 11 deletions(-) diff --git a/site/go.mod b/site/go.mod index 72c949ab76..7dd1ab90e6 100644 --- a/site/go.mod +++ b/site/go.mod @@ -2,4 +2,4 @@ module github.com/nginxinc/nginx-gateway-fabric/site go 1.21 -require github.com/nginxinc/nginx-hugo-theme v0.40.8 // indirect +require github.com/nginxinc/nginx-hugo-theme v0.41.0 // indirect diff --git a/site/go.sum b/site/go.sum index 009d2ae9c9..ea6bf85748 100644 --- a/site/go.sum +++ b/site/go.sum @@ -1,10 +1,2 @@ -github.com/nginxinc/nginx-hugo-theme v0.35.0 h1:7XB2GMy6qeJgKEJy9wOS3SYKYpfvLW3/H+UHRPLM4FU= -github.com/nginxinc/nginx-hugo-theme v0.35.0/go.mod h1:DPNgSS5QYxkjH/BfH4uPDiTfODqWJ50NKZdorguom8M= -github.com/nginxinc/nginx-hugo-theme v0.39.0 h1:P1hOPpityVUOM5OyIpQZa1UJyuUunGSmz0oZh/GYSJM= -github.com/nginxinc/nginx-hugo-theme v0.39.0/go.mod h1:DPNgSS5QYxkjH/BfH4uPDiTfODqWJ50NKZdorguom8M= -github.com/nginxinc/nginx-hugo-theme v0.40.0 h1:YP0I0+bRKcJ5WEb1s/OWcnlcvNvIcKscagJkCzsa+Vs= -github.com/nginxinc/nginx-hugo-theme v0.40.0/go.mod h1:DPNgSS5QYxkjH/BfH4uPDiTfODqWJ50NKZdorguom8M= -github.com/nginxinc/nginx-hugo-theme v0.40.1 h1:1Q94uFYegNvjvwDV1py9VlYmh62AF1gh1oPGqjNmtis= -github.com/nginxinc/nginx-hugo-theme v0.40.1/go.mod h1:DPNgSS5QYxkjH/BfH4uPDiTfODqWJ50NKZdorguom8M= -github.com/nginxinc/nginx-hugo-theme v0.40.8 h1:VtoSAtf9k67tI2jzbLRo0oFBAMHZBUPRh/xV4MYullI= -github.com/nginxinc/nginx-hugo-theme v0.40.8/go.mod h1:DPNgSS5QYxkjH/BfH4uPDiTfODqWJ50NKZdorguom8M= +github.com/nginxinc/nginx-hugo-theme v0.41.0 h1:uB9jC0Qk9i2CG63gScHxVHAEz1zyGoAdtY0Lcpkg1lI= +github.com/nginxinc/nginx-hugo-theme v0.41.0/go.mod h1:DPNgSS5QYxkjH/BfH4uPDiTfODqWJ50NKZdorguom8M=