Skip to content
This repository has been archived by the owner on Feb 22, 2022. It is now read-only.

[stable/kong] Options to install Prometheus plugin with ServiceMonitor #14346

Merged
merged 10 commits into from
Aug 19, 2019
Merged

[stable/kong] Options to install Prometheus plugin with ServiceMonitor #14346

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
72a7347
Options to install Prometheus plugin with ServiceMonitor
decayofmind May 30, 2019
c0ef24e
Merge branch 'master' into kong/prometheus-integration
decayofmind Jul 30, 2019
a40837d
Changes after code review
Aug 6, 2019
0e989c6
Merge branch 'master' into kong/prometheus-integration
decayofmind Aug 16, 2019
0e99b73
Removed ingressController and plugin dependency.
Aug 16, 2019
a45e32f
Version bumped
Aug 16, 2019
8b48986
Merge branch 'master' into kong/prometheus-integration
decayofmind Aug 19, 2019
06afe5e
Remove helm-hook annotations
Aug 19, 2019
ff2ea39
Merge branch 'master' into kong/prometheus-integration
decayofmind Aug 19, 2019
477c8d5
Remove trailing whitespace
Aug 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion stable/kong/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@ maintainers:
name: kong
sources:
- https://github.com/Kong/kong
version: 0.11.2
version: 0.13.0
appVersion: 1.1
55 changes: 41 additions & 14 deletions stable/kong/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,12 +105,12 @@ the value provided by you as opposed to constructing a listen variable
from fields like `proxy.http.containerPort` and `proxy.http.enabled`. This allows
you to be more prescriptive when defining listen directives.

**Note:** Overriding `env.proxy_listen` and `env.admin_listen` will potentially cause
`admin.containerPort`, `proxy.http.containerPort` and `proxy.tls.containerPort` to become out of sync,
**Note:** Overriding `env.proxy_listen` and `env.admin_listen` will potentially cause
`admin.containerPort`, `proxy.http.containerPort` and `proxy.tls.containerPort` to become out of sync,
and therefore must be updated accordingly.

I.E. updatating to `env.proxy_listen: 0.0.0.0:4444, 0.0.0.0:4443 ssl` will need
`proxy.http.containerPort: 4444` and `proxy.tls.containerPort: 4443` to be set in order
I.E. updatating to `env.proxy_listen: 0.0.0.0:4444, 0.0.0.0:4443 ssl` will need
`proxy.http.containerPort: 4444` and `proxy.tls.containerPort: 4443` to be set in order
for the service definition to work properly.

### Kong-specific parameters
Expand Down Expand Up @@ -157,7 +157,7 @@ kong:
key: kong
name: postgres
```


For complete list of Kong configurations please check https://getkong.org/docs/latest/configuration/.

Expand Down Expand Up @@ -218,12 +218,39 @@ You can can learn about kong ingress custom resource definitions here:

- [https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/custom-resources.md]()

| Parameter | Description | Default |
| --------------- | ----------------------------------------- | ---------------------------------------------------------------------------- |
| enabled | Deploy the ingress controller, rbac and crd | false |
| replicaCount | Number of desired ingress controllers | 1 |
| image.repository | Docker image with the ingress controller | kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller |
| image.tag | Version of the ingress controller | 0.2.0 |
| readinessProbe | Kong ingress controllers readiness probe | |
| livenessProbe | Kong ingress controllers liveness probe | |
| ingressClass | The ingress-class value for controller | nginx
#### Expose metrics in Prometheus format

Kong can expose metrics in Prometheus format if prometheus plugin is enabled.
Presence of Kong Ingress Controller allows you to enable this plugin, using
KongPlugin custom resource during the chart installation:

```bash
helm install stable/kong --set ingressController.enabled=true \
--set ingressController.metrics.enabled=true --name kong --namespace kong
```

Optionaly you can install a ServiceMonitor for Prometheus Operator:

```bash
helm install stable/kong --set ingressController.enabled=true \
--set ingressController.metrics.enabled=true \
--set ingressController.metrics.serviceMonitor.create=true \
--name kong --namespace kong
```


| Parameter | Description | Default |
| --------------- | ------------------------------------------ | ---------------------------------------------------------------------------- |
| enabled | Deploy the ingress controller, rbac and crd | false |
| replicaCount | Number of desired ingress controllers | 1 |
| image.repository | Docker image with the ingress controller | kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller |
| image.tag | Version of the ingress controller | 0.4.0 |
| readinessProbe | Kong ingress controllers readiness probe | |
| livenessProbe | Kong ingress controllers liveness probe | |
| ingressClass | The ingress-class value for controller | kong |
| installCRDs | Install Kong ingress controllers custom resource definitions | true |
| cleanUpCRDs | Delete CRDs with the release (useful for CI scenarios) | false |
| metrics.enabled | Enable Kong Prometheus plugin as KongPlugin (requires ingressController to be enabled) | false |
| metrics.serviceMonitor.create | Create ServiceMonitor for Prometheus Operator | false |
| metrics.serviceMonitor.interval | Scrapping interval | 10s |
| metrics.serviceMonitor.namespace | Where to create ServiceMonitor | |
1 change: 1 addition & 0 deletions stable/kong/ci/dbless.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# CI test for testing dbless deployment
ingressController:
enabled: true
cleanUpCRDs: true
env:
database: "off"
postgresql:
Expand Down
1 change: 1 addition & 0 deletions stable/kong/ci/ingressController-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# CI test for Ingress controller basic installation
ingressController:
enabled: true
cleanUpCRDs: true
40 changes: 40 additions & 0 deletions stable/kong/templates/cleanup-crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{{- if .Values.ingressController.cleanUpCRDs }}
apiVersion: batch/v1
decayofmind marked this conversation as resolved.
Show resolved Hide resolved
kind: Job
metadata:
name: {{ template "kong.fullname" . }}-cleanup-crds
namespace: {{ .Release.Namespace }}
annotations:
"helm.sh/hook": pre-delete
"helm.sh/hook-weight": "3"
"helm.sh/hook-delete-policy": hook-succeeded
labels:
app: {{ template "kong.name" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
spec:
template:
metadata:
name: kong-cleanup-crds
spec:
serviceAccountName: {{ include "kong.serviceAccountName" . }}
containers:
- name: kubectl
image: docker.io/bitnami/kubectl:1.14.1
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- >
kubectl delete kongconsumers --all;
kubectl delete kongcredentials --all;
kubectl delete kongingresses --all;
kubectl delete kongplugins --all;
sleep 10;
kubectl delete crd kongconsumers.configuration.konghq.com;
kubectl delete crd kongcredentials.configuration.konghq.com;
kubectl delete crd kongingresses.configuration.konghq.com;
kubectl delete crd kongplugins.configuration.konghq.com;
restartPolicy: OnFailure
{{- end }}
5 changes: 4 additions & 1 deletion stable/kong/templates/crd-kongconsumer.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{- if and .Values.ingressController.enabled .Values.ingressController.installCRDs -}}
{{- if and .Release.IsInstall .Values.ingressController.enabled .Values.ingressController.installCRDs -}}
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
Expand All @@ -8,6 +8,9 @@ metadata:
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
annotations:
"helm.sh/hook": crd-install
"helm.sh/hook-delete-policy": "before-hook-creation"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe these changes can go away once KongPlugin resource is removed.

spec:
group: configuration.konghq.com
version: v1
Expand Down
5 changes: 4 additions & 1 deletion stable/kong/templates/crd-kongcredential.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{- if and .Values.ingressController.enabled .Values.ingressController.installCRDs -}}
{{- if and .Release.IsInstall .Values.ingressController.enabled .Values.ingressController.installCRDs -}}
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
Expand All @@ -8,6 +8,9 @@ metadata:
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
annotations:
"helm.sh/hook": crd-install
"helm.sh/hook-delete-policy": "before-hook-creation"
spec:
group: configuration.konghq.com
version: v1
Expand Down
5 changes: 4 additions & 1 deletion stable/kong/templates/crd-kongingress.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{- if and .Values.ingressController.enabled .Values.ingressController.installCRDs -}}
{{- if and .Release.IsInstall .Values.ingressController.enabled .Values.ingressController.installCRDs -}}
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
Expand All @@ -8,6 +8,9 @@ metadata:
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
annotations:
"helm.sh/hook": crd-install
"helm.sh/hook-delete-policy": "before-hook-creation"
spec:
group: configuration.konghq.com
version: v1
Expand Down
5 changes: 4 additions & 1 deletion stable/kong/templates/crd-kongplugins.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{- if and .Values.ingressController.enabled .Values.ingressController.installCRDs -}}
{{- if and .Release.IsInstall .Values.ingressController.enabled .Values.ingressController.installCRDs -}}
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
Expand All @@ -8,6 +8,9 @@ metadata:
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
annotations:
"helm.sh/hook": crd-install
"helm.sh/hook-delete-policy": "before-hook-creation"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these changes to CRDs can be removed since the Prometheus plugin is not being automatically created.

Another problem that I see here:
If multiple Kong Ingress Controllers are being installed one after the other, then this will incorrectly delete the CRD first, and then re-install the CRD. This will lead to deletion of all the custom resources that were present in the cluster before the installation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@decayofmind Could you address this comment?
Thanks!

spec:
group: configuration.konghq.com
version: v1
Expand Down
13 changes: 13 additions & 0 deletions stable/kong/templates/prometheus-plugin.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{{- if and .Values.ingressController.enabled .Values.ingressController.metrics.enabled }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said in a review before, let's remove this.
The user can create an additional KongPlugin resource if the need be after the Kong Ingress Controller is installed.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do note that Kong will expose some basic observability stats even if this plugin is not enabled.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, let it be as you're suggesting, I've got the point :)

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: prometheus
decayofmind marked this conversation as resolved.
Show resolved Hide resolved
labels:
app: {{ template "kong.name" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
global: "true"
plugin: prometheus
{{- end }}
33 changes: 33 additions & 0 deletions stable/kong/templates/servicemonitor.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{{- if and ( .Capabilities.APIVersions.Has "monitoring.coreos.com/v1" ) .Values.ingressController.enabled .Values.ingressController.metrics.serviceMonitor.create }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ template "kong.fullname" . }}
{{- if .Values.ingressController.metrics.serviceMonitor.namespace }}
namespace: {{ .Values.ingressController.metrics.serviceMonitor.namespace }}
{{- end }}
labels:
app: {{ template "kong.name" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
spec:
endpoints:
- port: kong-admin
decayofmind marked this conversation as resolved.
Show resolved Hide resolved
{{- if .Values.ingressController.metrics.serviceMonitor.interval }}
interval: {{ .Values.ingressController.metrics.serviceMonitor.interval }}
{{- end }}
{{- if .Values.admin.useTLS }}
scheme: https
tlsConfig:
insecureSkipVerify: true
{{- end }}
jobLabel: {{ .Release.Name }}
namespaceSelector:
matchNames:
- {{ .Release.Namespace }}
selector:
matchLabels:
app: {{ template "kong.name" . }}
release: {{ .Release.Name }}
{{- end }}
11 changes: 11 additions & 0 deletions stable/kong/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,7 @@ ingressController:
timeoutSeconds: 5

installCRDs: true
cleanUpCRDs: false

rbac:
# Specifies whether RBAC resources should be created
Expand All @@ -213,3 +214,13 @@ ingressController:
name:

ingressClass: kong

metrics:
# Enables Kong Prometheus plugin as KongPlugin object
enabled: false
serviceMonitor:
# Specifies whether ServiceMonitor for Prometheus operator should be created
create: false
# interval: 10s
# Specifies namespace, where ServiceMonitor should be installed
# namespace: monitoring