Skip to content

Commit

Permalink
feat(helm chart): Add metrics-server hardening options
Browse files Browse the repository at this point in the history
  • Loading branch information
@mkilchhofer
mkilchhofer committed Jul 14, 2023
1 parent 796fc0f commit 1a6b939
Show file tree
Hide file tree
Showing 8 changed files with 293 additions and 2 deletions.
110 changes: 110 additions & 0 deletions charts/metrics-server/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,113 @@ The following table lists the configurable parameters of the _Metrics Server_ ch
| `topologySpreadConstraints` | Pod Topology Spread Constraints. | `[]` |
| `deploymentAnnotations` | Annotations to add to the deployment. | `{}` |
| `schedulerName` | scheduler to set to the deployment. | `""` |
| `tls.type` | TLS option to use. Either use `metrics-server` for self-signed certificates, `secret`, `existingSecret` or `cert-manager` | `"metrics-server"` |
| `tls.certManager.clusterDomain` | Kubernetes cluster domain. Used to configure Subject Alt Names for the certificate | `"cluster.local"` |
| `tls.certManager.addInjectorAnnotations` | Automatically add the cert-manager.io/inject-ca-from annotation to the APIService resource. | `true` |
| `tls.certManager.existingIssuer.enabled` | Use an existing cert-manager issuer | `false` |
| `tls.certManager.existingIssuer.kind` | Kind of the existing cert-manager issuer | `"Issuer"` |
| `tls.certManager.existingIssuer.name` | Name of the existing cert-manager issuer | `"my-issuer"` |
| `tls.certManager.duration` | Set the requested duration (i.e. lifetime) of the Certificate. | `""` |
| `tls.certManager.renewBefore` | How long before the currently issued certificate’s expiry cert-manager should renew the certificate. | `""` |
| `tls.certManager.annotations` | Add extra annotations to the Certificate resource | `{}` |
| `tls.certManager.labels` | Add extra labels to the Certificate resource | `{}` |
| `tls.existingSecret.name` | Name of the existing Secret to use for TLS | `""` |
| `tls.secret.annotations` | Add extra annotations to the Secret resource | `{}` |
| `tls.secret.labels` | Add extra labels to the Secret resource | `{}` |
| `tls.secret.crt` | The server certificate to use for metrics-server. Use PEM format | `""` |
| `tls.secret.key` | The private key of the certificate to use for metrics-server. Use PEM format. | `""` |
| `extraObjects` | List of extra manifests to deploy. Will be passed through `tpl` to support templating | `[]` |

## Hardening metrics-server

By default, metrics-server is using a self-signed certificate which is generated during startup. The APIservice is registered with `.spec.insecureSkipTLSVerify` set to `true` as you can see here:

```yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta1.metrics.k8s.io
spec:
#..
insecureSkipTLSVerify: true # <-- see here
service:
name: metrics-server
#..
```

To harden metrics-server, you have these 3 options described in the following section.

### Option 1: Use cert-manager

> **Requirement:** cert-manager needs to be installed before you install metrics-server
To use this method, please setup your values file like this:

```yaml
apiService:
insecureSkipTLSVerify: false
tls:
type: cert-manager
```
There are other optional parameters, if you want to customize the behavior of the certificate even more.
### Option 2: Provide certificate data
You can use an arbitrary PKI solution and generate a certificate for metrics-server. You need the following data in PEM format:
- The CA certificate
- The server certificate which is issued with 3 Subject Alt Names:
- `metrics-server.<namespace>`
- `metrics-server.<namespace>.svc`
- `metrics-server.<namespace>.svc.cluster.local`
- The server key file

To use this method, please setup your values file like this:

```yaml
apiService:
insecureSkipTLSVerify: false
caBundle: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls:
type: secret
secret:
crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
key: |
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
```

### Option 3: Use existing Secret

This option allows you to reuse an existing Secret. This Secrets can have an arbitrary origin, e.g.

- Created via kubectl / Terraform / etc.
- Synced from a secrets management solution like AWS SecretsManager, HashiCorp Vault, etc.

You still need to pass the CA certificate to ensure proper configuration of the `APIservice` resource,
but the sensitive information (the private key) can be read via Secret. To use this method, please setup your values file like this:

```yaml
apiService:
insecureSkipTLSVerify: false
caBundle:
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls:
type: existingSecret
existingSecret:
name: metrics-server-existing
```
8 changes: 8 additions & 0 deletions charts/metrics-server/ci/tls-certManager-values.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
args:
- --kubelet-insecure-tls

apiService:
insecureSkipTLSVerify: false

tls:
type: cert-manager
9 changes: 7 additions & 2 deletions charts/metrics-server/templates/apiservice.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,14 @@ metadata:
name: v1beta1.metrics.k8s.io
labels:
{{- include "metrics-server.labels" . | nindent 4 }}
{{- with .Values.apiService.annotations }}
{{- if or .Values.apiService.annotations .Values.tls.certManager.addInjectorAnnotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- if .Values.tls.certManager.addInjectorAnnotations }}
cert-manager.io/inject-ca-from: {{ .Release.Namespace }}/{{ include "metrics-server.fullname" . }}
{{- end }}
{{- with .Values.apiService.annotations }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}
spec:
{{- with .Values.apiService.caBundle }}
Expand Down
47 changes: 47 additions & 0 deletions charts/metrics-server/templates/certificate.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
{{- if eq .Values.tls.type "cert-manager" }}
{{- if not .Values.tls.certManager.existingIssuer.enabled }}
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
annotations:
{{- toYaml .Values.additionalAnnotations | nindent 4 }}
name: {{ include "metrics-server.fullname" . }}-issuer
namespace: {{ .Release.Namespace }}
spec:
selfSigned: {}
{{- end }}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: {{ include "metrics-server.fullname" . }}
namespace: {{ .Release.Namespace }}
spec:
commonName: {{ include "metrics-server.fullname" . }}
dnsNames:
- {{ include "metrics-server.fullname" . }}.{{ .Release.Namespace }}
- {{ include "metrics-server.fullname" . }}.{{ .Release.Namespace }}.svc
- {{ include "metrics-server.fullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.tls.certManager.clusterDomain }}
secretName: {{ include "metrics-server.fullname" . }}
usages:
- server auth
- client auth
privateKey:
algorithm: RSA
size: 2048
{{- with .Values.tls.certManager.duration }}
duration: {{ . }}
{{- end }}
{{- with .Values.tls.certManager.renewBefore }}
renewBefore: {{ . }}
{{- end }}
issuerRef:
{{- if .Values.tls.certManager.existingIssuer.enabled }}
name: {{ .Values.tls.certManager.existingIssuer.name }}
kind: {{ .Values.tls.certManager.existingIssuer.kind }}
{{- else }}
name: {{ include "metrics-server.fullname" . }}-issuer
kind: Issuer
{{- end }}
group: cert-manager.io
{{- end }}
18 changes: 18 additions & 0 deletions charts/metrics-server/templates/deployment.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,10 @@ spec:
{{- if .Values.metrics.enabled }}
- --authorization-always-allow-paths=/metrics
{{- end }}
{{- if ne .Values.tls.type "metrics-server" }}
- --tls-cert-file=/tmp/tls-certs/tls.crt
- --tls-private-key-file=/tmp/tls-certs/tls.key
{{- end }}
{{- range .Values.args }}
- {{ . }}
{{- end }}
Expand All @@ -80,6 +84,11 @@ spec:
volumeMounts:
- name: tmp
mountPath: /tmp
{{- if ne .Values.tls.type "metrics-server" }}
- mountPath: /tmp/tls-certs
name: certs
readOnly: true
{{- end }}
{{- with .Values.extraVolumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
Expand Down Expand Up @@ -126,6 +135,15 @@ spec:
configMap:
name: {{ include "metrics-server.addonResizer.configMap" . }}
{{- end }}
{{- if ne .Values.tls.type "metrics-server" }}
- name: certs
secret:
{{- if and (eq .Values.tls.type "existingSecret") .Values.tls.existingSecret.name }}
secretName: {{ .Values.tls.existingSecret.name }}
{{- else }}
secretName: {{ .Values.tls.existingSecret.name }}{{ include "metrics-server.fullname" . }}
{{- end }}
{{- end }}
{{- with .Values.extraVolumes }}
{{- toYaml . | nindent 8 }}
{{- end }}
Expand Down
4 changes: 4 additions & 0 deletions charts/metrics-server/templates/extra-manifests.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{{ range .Values.extraObjects }}
---
{{ tpl (toYaml .) $ }}
{{ end }}
19 changes: 19 additions & 0 deletions charts/metrics-server/templates/secret.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{{- if eq .Values.tls.type "secret" }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "metrics-server.fullname" . }}
{{- with .Values.tls.secret.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
labels:
{{- include "metrics-server.labels" . | nindent 4 }}
{{- with .Values.tls.secret.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
type: Opaque
data:
tls.crt: {{ .Values.tls.secret.crt | b64enc | quote }}
tls.key: {{ .Values.tls.secret.key | b64enc | quote }}
{{- end }}
80 changes: 80 additions & 0 deletions charts/metrics-server/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -174,3 +174,83 @@ topologySpreadConstraints: []
deploymentAnnotations: {}

schedulerName: ""

tls:
# Either use `metrics-server`, `secret`, `existingSecret` or `cert-manager`. Details:
# `metrics-server` : Metrics-server will generate self-signed certs
# `secret` : Create a new secret with user-provided key and certificate
# `existingSecret` : Reuse an existing secret. No new secret will be created
# `cert-manager` : Use cert-manager.io to create and maintain the certificate for you
type: "metrics-server"

certManager:
# Kubernetes cluster domain. Used to configure Subject Alt Names for the certificate
clusterDomain: cluster.local
# Automatically add the cert-manager.io/inject-ca-from annotation to the APIService resource.
# See https://cert-manager.io/docs/concepts/ca-injector
addInjectorAnnotations: true
existingIssuer:
# Use an existing cert-manager issuer
enabled: false
# Kind of the existing cert-manager issuer
kind: "Issuer"
# Name of the existing cert-manager issuer
name: "my-issuer"
# Set the requested duration (i.e. lifetime) of the Certificate.
# See https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.CertificateSpec
duration: ""
# How long before the currently issued certificate’s expiry cert-manager should renew the certificate.
# See https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.CertificateSpec
renewBefore: ""
# Add extra annotations to the Certificate resource
annotations: {}
# Add extra labels to the Certificate resource
labels: {}

existingSecret:
# Name of the existing Secret to use for TLS
name: ""

secret:
# Add extra annotations to the Secret resource
annotations: {}
# Add extra labels to the Secret resource
labels: {}
# The server certificate to use for metrics-server. Use PEM format
crt: ""
# The private key of the certificate to use for metrics-server. Use PEM format.
key: ""

# List of extra manifests to deploy. Will be passed through `tpl` to support templating
extraObjects: []
# - apiVersion: external-secrets.io/v1beta1
# kind: ExternalSecret
# metadata:
# name: my-external-secret
# namespace: '{{ .Release.Namespace }}'
# spec:
# secretStoreRef:
# kind: ClusterSecretStore
# name: my-secret-store
# target:
# name: my-kubernetes-secret
# data:
# - secretKey: secretKey
# remoteRef:
# key: /path/to/my-secret

# - apiVersion: secrets.hashicorp.com/v1beta1
# kind: VaultStaticSecret
# metadata:
# name: my-external-vault-secret
# namespace: '{{ .Release.Namespace }}'
# spec:
# type: kv-v2
# mount: kvv2
# path: webapp/config
# destination:
# name: secretkv
# create: true
# refreshAfter: 30s
# refreshAfter: 30s
# vaultAuthRef: static-auth

0 comments on commit 1a6b939

Please sign in to comment.