Change certificates expiration from 14 to 13 days (#758)

* Change certificates expiration from 14 to 13 days

The reason for this is a bug in cert manager. The Certificates we have
contain the CA that they've been signed with. When a CA has been
renewed, the Certificates that use it will not be updated until they
themselves are renewed. This causes an issue because the Certificate (in
KIAM's case) will renew a few hours after the CA. This is enough time
for the CA in the kiam certificate to expire and cause a flurry of
alerts.
This has mainly been observed with KIAM, but we decided to change all
alerts.

See cert-manager/cert-manager#5851

* Update CHANGELOG

CHANGELOG.md

@@ -10,6 +10,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Changed
 
 - Reduce PrometheusRuleFailures interval to 5m
+- Reduce CertificateWillExpireInLessThanTwoWeeks period from 14 days to 13
+  days. This accounts for a [bug in cert-manager](https://github.com/cert-manager/cert-manager/issues/5851) and gives
+  the certificate an extra day to renew without paging.
 
 ## [2.98.0] - 2023-05-10
 

helm/prometheus-rules/templates/alerting-rules/certificate.all.rules.yml

@@ -14,7 +14,7 @@ spec:
       annotations:
         description: '{{`Kiam Certificate stored in Secret {{ $labels.namespace }}/{{ $labels.name }} on {{ $labels.cluster_id }} will expire in less than two weeks.`}}'
         opsrecipe: managed-app-cert-manager/certificate-secret-will-expire-in-less-than-two-weeks/
-      expr: (cert_exporter_secret_not_after{name=~"kiam.*"} - time()) < 2 * 7 * 24 * 60 * 60
+      expr: (cert_exporter_secret_not_after{name=~"kiam.*"} - time()) < 13 * 24 * 60 * 60
       for: 5m
       labels:
         area: kaas
@@ -26,7 +26,7 @@ spec:
       annotations:
         description: '{{`IRSA Pod Identity Webhook Certificate stored in Secret {{ $labels.namespace }}/{{ $labels.name }} on {{ $labels.cluster_id }} will expire in less than two weeks.`}}'
         opsrecipe: managed-app-cert-manager/certificate-secret-will-expire-in-less-than-two-weeks/
-      expr: (cert_exporter_secret_not_after{name=~"aws-pod-identity-webhook.*"} - time()) < 2 * 7 * 24 * 60 * 60
+      expr: (cert_exporter_secret_not_after{name=~"aws-pod-identity-webhook.*"} - time()) < 13 * 24 * 60 * 60
       for: 5m
       labels:
         area: kaas
@@ -49,7 +49,7 @@ spec:
       annotations:
         description: '{{`Certificate stored in Secret {{ $labels.namespace }}/{{ $labels.name }} on {{ $labels.cluster_id }} will expire in less than two weeks.`}}'
         opsrecipe: managed-app-cert-manager/certificate-secret-will-expire-in-less-than-two-weeks/
-      expr: (cert_exporter_secret_not_after{name!~"kiam.*|.*kyverno.*",cluster_type="management_cluster"} - time()) < 2 * 7 * 24 * 60 * 60
+      expr: (cert_exporter_secret_not_after{name!~"kiam.*|.*kyverno.*",cluster_type="management_cluster"} - time()) < 13 * 24 * 60 * 60
       labels:
         area: kaas
         cancel_if_outside_working_hours: "true"
@@ -60,7 +60,7 @@ spec:
       annotations:
         description: '{{`Certificate CR {{ $labels.namespace }}/{{ $labels.name }} on {{ $labels.cluster_id }} will expire in less than two weeks.`}}'
         opsrecipe: managed-app-cert-manager/certificate-secret-will-expire-in-less-than-two-weeks/
-      expr: (cert_exporter_certificate_cr_not_after{managed_issuer="true",namespace=~"kube-system|giantswarm|monitoring"} - time()) < 2 * 7 * 24 * 60 * 60
+      expr: (cert_exporter_certificate_cr_not_after{managed_issuer="true",namespace=~"kube-system|giantswarm|monitoring"} - time()) < 13 * 24 * 60 * 60
       labels:
         area: kaas
         cancel_if_outside_working_hours: "true"
@@ -71,7 +71,7 @@ spec:
       annotations:
         description: '{{`Certificate CR {{ $labels.namespace }}/{{ $labels.name }} on {{ $labels.cluster_id }} will expire in less than two weeks.`}}'
         opsrecipe: managed-app-cert-manager/certificate-secret-will-expire-in-less-than-two-weeks/
-      expr: (cert_exporter_certificate_cr_not_after{managed_issuer="true",cluster_type="workload_cluster",namespace!~"kube-system|giantswarm|monitoring"} - time()) < 2 * 7 * 24 * 60 * 60
+      expr: (cert_exporter_certificate_cr_not_after{managed_issuer="true",cluster_type="workload_cluster",namespace!~"kube-system|giantswarm|monitoring"} - time()) < 13 * 24 * 60 * 60
       labels:
         area: kaas
         cancel_if_outside_working_hours: "true"