feat(security): implement tls block for grafana external

[aboulay-numspot]

Aim of the merge request

Implement the possibility to use a custom CA and use a client certificate (tls.crt and tls.key) when declaring an external Grafana resource.

This will improve security of the solution because, after analysis, it seems to be in insecureSkipVerify per default in every tls connection.

Breaking change

Currently, every call is in insecureSkipVerify by default regardless if they are using https or not. This MR provides the possibility to use the tls block to make https connection and specify if the connection is insecure or not. If the tls block is not specified, http request will be sent instead.

Related documents

Design document: Add TLS management block in Grafana CR External block

Testing process

To test the tls with a self-signed certificate, we will need to deploy a grafana service (without the operator) with a nginx configured has a https reverse proxy.

Homemade testing process (click to open)

To do this, we will create a self-signed certifcate:

• First create the config file crt.conf for the certificate:

[ req ]
prompt = no
distinguished_name = dn
req_extensions = req_ext

[ dn ]
CN = endpoint.grafana-ext.svc.cluster.local
emailAddress = [email protected]
O = Test
L = Courbevoie
ST = France
C = FR

[ req_ext ]
subjectAltName = DNS: endpoint.grafana-ext.svc.cluster.local

• Then, create the v3.ext file:

subjectAltName         = DNS:endpoint.grafana-ext.svc.cluster.local

• Then generate the certifcate and sign it with the following command:

# generate the ca.key
openssl genrsa -out ca.key 4096

# generate the ca.crt from the ca.key
openssl req -x509 -new -nodes -key ca.key -sha256 -days 1024 -out ca.crt 

# generate the csr based on a configuration. Required to get the SAN into the certificate
openssl req -new -config crt.conf -key tls.key -out tls.csr

# generate the crt based on the csr and sign it with the CA
openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt -days 500 -sha256 -extfile v3.ext

Then, we will declare the manifest for k8s resources:

• grafana related resources:

---
apiVersion: v1
kind: Namespace
metadata:
  name: grafana-ext

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana-ext
  namespace: grafana-ext
  labels:
    app: grafana-ext
spec:
  selector:
    matchLabels:
      app: grafana-ext
  template:
    metadata:
      labels:
        app: grafana-ext
    spec:
      containers:
        - name: grafana
          image: grafana/grafana:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          resources:
            requests:
              cpu: 250m
              memory: 750Mi

---
apiVersion: v1
kind: Service
metadata:
  name: grafana-ext
  namespace: grafana-ext
spec:
  ports:
    - port: 3000
      protocol: TCP
      targetPort: http-grafana
  selector:
    app: grafana-ext

• resource related of the nginx (in the same namespace)

--
apiVersion: v1
kind: Secret
metadata:
  name: tls-certificate
  namespace: grafana-ext
data:
  tls.crt: <tls.crt content base64 encoded>
  tls.key: <tls.key content base64 encoded>
  ca.crt: <ca.crt content base64 encoded>
type: Opaque

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: grafana-ext
data:
  nginx.conf: |
    events {}
    http {
      server {
        listen       443 ssl;
        ssl_certificate     /etc/nginx/certs/tls.crt;
        ssl_certificate_key /etc/nginx/certs/tls.key;
        server_name  endpoint.grafana-ext.svc.cluster.local;
        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;

        location / {
            proxy_pass http://grafana-ext:3000/;
        }
      }
    }

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: grafana-ext
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 443
        volumeMounts:
        - mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
          name: nginx-conf
        - mountPath: /etc/nginx/certs
          name: tls-certificate
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-config
          items:
            - key: nginx.conf
              path: nginx.conf
      - name: tls-certificate
        secret:
          secretName: tls-certificate

---
apiVersion: v1
kind: Service
metadata:
  name: endpoint
  namespace: grafana-ext
spec:
  ports:
  - port: 443
    targetPort: 443
  selector:
    app: nginx

• the tls secret for the Grafana CR

---
apiVersion: v1
kind: Namespace
metadata:
  name: consumer

--
apiVersion: v1
kind: Secret
metadata:
  name: tls-certificate
  namespace: consumer
data:
  tls.crt: <tls.crt content base64 encoded>
  tls.key: <tls.key content base64 encoded>
  ca.crt: <ca.crt content base64 encoded>
type: Opaque

And finaly, the Grafana CR

---
apiVersion: v1
kind: Secret
metadata:
  name: grafana-admin-credentials
  namespace: consumer
data:
  GF_SECURITY_ADMIN_PASSWORD: YWRtaW4=
  GF_SECURITY_ADMIN_USER: YWRtaW4=
type: Opaque

---
# Use the same grafana instance that we just created, notice the ingress config
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: external-grafana
  namespace: consumer
  labels:
    dashboards: "external-grafana"
spec:
  external:
    url: https://endpoint.grafana-ext.svc.cluster.local
    adminPassword:
      name: grafana-admin-credentials
      key: GF_SECURITY_ADMIN_PASSWORD
    adminUser:
      name: grafana-admin-credentials
      key: GF_SECURITY_ADMIN_USER
    tls:
      insecureSkipVerify: true
      #certSecretRef:
      #  name: tls-certificate

• a dashboard to ensure the connection is working well:

---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: grafana-root
  namespace: consumer
  labels:
    dashboards: "external-grafana"
spec:
  instanceSelector:
    matchLabels:
      dashboards: "external-grafana"
  json: |
    {
      "id": null,
      "title": "Simple Dashboard",
      "tags": [],
      "style": "dark",
      "timezone": "browser",
      "editable": true,
      "hideControls": false,
      "graphTooltip": 1,
      "panels": [],
      "time": {
        "from": "now-6h",
        "to": "now"
      },
      "timepicker": {
        "time_options": [],
        "refresh_intervals": []
      },
      "templating": {
        "list": []
      },
      "annotations": {
        "list": []
      },
      "refresh": "5s",
      "schemaVersion": 17,
      "version": 0,
      "links": []
    }

Then, initialize the kind cluster locally:

make start-kind

export KUBECONFIG=$HOME/.kube/kind-grafana-operator

kubectl delete grafanadashboard/grafana -n grafana-crds; kubectl delete grafanadashboard/grafana; kubectl delete grafana/grafana

# Here I recommend to add a tag to specify the image

make ko-build-kind

make deploy

kubectl edit deploy/grafana-operator-controller-manager-v5 -n grafana-operator-system
# Change the line
# >        image: ghcr.io/grafana/grafana-operator:v5.11.0
# >        imagePullPolicy: Always
# <        image: ko.local/grafana/grafana-operator
# <        imagePullPolicy: IfNotPresent

# Deploy the manifest in this order
kubectl apply -f consumer.yml
kubectl apply -f nginx.yml
kubectl apply -f consumer-tls.yml
kubectl apply -f consumer.yml
kubectl apply -f dashboard.yml

You can now start playing with the parameter from the Grafana CR to test the certificate connectivity.

Additional comments

The behavior of using insecure https connection have not been changed for the rest of the system (grafana_com_fetcher to retrieve dashboard or url_fetcher). The NewHTTPClient has been kept because the Grafana CR's getVersion method use an api path not declared into the grafana-openapi-client (GET /api/frontend/settings).

Just a contextual thing to finish this (too log message), I'll be in holiday for two weeks so the review will probably take a bunch of time before been handled.

[theSuess]

Thanks for the PR! Looks great at first glance. It'll take me some time to review, but I'll try to fit it in next week

[aboulay-numspot]

@theSuess I let you review these changes to ensure everything is fine for you

[theSuess]

One last nitpick and then I'm good to merge this!