This guide will run through installing and using istio-csr from scratch. We'll use kind to create a new cluster locally in Docker, but this guide should work on any cluster as long as the relevant Istio Platform Setup has been performed.
Note that if you're following the Platform Setup guide for OpenShift, do not run the istioctl install command listed in that guide; we'll run our own command later.
You'll need the following tools installed on your machine:
In addition, Istio must not already be installed in your cluster. Installing istio-csr after Istio is not supported.
Kind will automatically set up kubectl to point to the newly created cluster.
We install cert-manager using helm here, but if you've got a preferred method you can install in any way.
kind create cluster --image=docker.io/kindest/node:v1.22.4
# Helm setup
helm repo add jetstack https://charts.jetstack.io
helm repo update
# install cert-manager CRDs
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.crds.yaml
# install cert-manager; this might take a little time
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.6.1
# We need this namespace to exist since our cert will be placed there
kubectl create namespace istio-systemAn Issuer tells cert-manager how to issue certificates; we'll create a self-signed root CA in our cluster because it's really simple to configure.
The approach of using a locally generated root certificate would work in a production deployment too, but there are also several other issuers in cert-manager which could be used. Note that the ACME issuer will not work, since it can't add the required fields to issued certificates.
There are also some comments on the example-issuer providing a little more detail. Note also that this guide only uses Issuers and not ClusterIssuers - using a ClusterIssuer isn't a drop-in replacement, and in any case we recommend that production deployments use Issuers for easier access controls and scoping.
kubectl apply -f https://raw.githubusercontent.com/cert-manager/istio-csr/main/docs/example-issuer.yamlWhile it's possible to configure Istio such that it can automatically "discover" the root CA, this can be dangerous in some specific scenarios involving other security holes, enabling signer hijacking attacks.
As such, we'll export our Root CA and configure Istio later using that static cert.
# Export our cert from the secret it's stored in, and base64 decode to get the PEM data.
kubectl get -n istio-system secret istio-ca -ogo-template='{{index .data "tls.crt"}}' | base64 -d > ca.pem
# Out of interest, we can check out what our CA looks like
openssl x509 -in ca.pem -noout -text
# Add our CA to a secret
kubectl create secret generic -n cert-manager istio-root-ca --from-file=ca.pem=ca.pemistio-csr is best installed via Helm, and it should be simple and quick to install. There are a bunch of other configuration options for the helm chart, which you can check out here.
helm repo add jetstack https://charts.jetstack.io
helm repo update
# We set a few helm template values so we can point at our static root CA
helm install -n cert-manager cert-manager-istio-csr jetstack/cert-manager-istio-csr \
--set "app.tls.rootCAFile=/var/run/secrets/istio-csr/ca.pem" \
--set "volumeMounts[0].name=root-ca" \
--set "volumeMounts[0].mountPath=/var/run/secrets/istio-csr" \
--set "volumes[0].name=root-ca" \
--set "volumes[0].secret.secretName=istio-root-ca"
# Check to see that the istio-csr pod is running and ready
kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-aaaaaaaaaa-11111 1/1 Running 0 9m46s
cert-manager-cainjector-aaaaaaaaaa-22222 1/1 Running 0 9m46s
cert-manager-istio-csr-bbbbbbbbbb-00000 1/1 Running 0 63s
cert-manager-webhook-aaaaaaaaa-33333 1/1 Running 0 9m46sIf you're not running on kind, you may need to do some additional setup tasks before installing Istio.
We use the istioctl CLI to install Istio, configured using a custom IstioOperator manifest.
The custom manifest does the following:
- Disables the CA server in istiod,
- Ensures that Istio workloads request certificates from istio-csr,
- Ensures that the istiod certificates and keys are mounted from the Certificate created when installing istio-csr.
First we download our demo manifest and then we apply it.
curl -sSL https://raw.githubusercontent.com/cert-manager/istio-csr/main/docs/istio-config-getting-started.yaml > istio-install-config.yamlYou may wish to inspect and tweak istio-install-config.yaml if you know what you're doing,
but this manifest should work for example purposes as-is.
If you set a custom app.tls.trustDomain when installing istio-csr via helm earlier, you'll need to ensure that
value is repeated in istio-install-config.yaml.
This final command will install Istio; the exact command you need might vary on different platforms, and will certainly vary on OpenShift.
# This takes a little time to complete
istioctl install -f istio-install-config.yaml
# If you're on OpenShift, you need a different profile:
# istioctl install --set profile=openshift -f istio-install-config.yamlThe following steps are option but can be followed to validate everything is hooked correctly:
- Deploy a sample application & watch for
certificaterequests.cert-manager.ioresources - Verify
cert-managerlogs for newcertificaterequestsand responses - Verify the CA Endpoint being used in a
istio-proxysidecar container - Using
istioctlto fetch the certificate info for theistio-proxycontainer
To see this all in action, lets deploy a very simple sample application from the istio samples.
First set some environment variables whose values could be changed if needed:
# Set namespace for sample application
export NAMESPACE=default
# Set env var for the value of the app label in manifests
export APP=httpbin
# Grab the installed version of istio
export ISTIO_VERSION=$(istioctl version -o json | jq -r '.meshVersion[0].Info.version')We use the default namespace for simplicity, so let's label the namespace for istio injection:
kubectl label namespace $NAMESPACE istio-injection=enabled --overwriteIn a separate terminal you should now follow the logs for cert-manager:
kubectl logs -n cert-manager $(kubectl get pods -n cert-manager -o jsonpath='{.items..metadata.name}' --selector app=cert-manager) --since 2m -fIn another separate terminal, lets watch the istio-system namespace for certificaterequests:
kubectl get certificaterequests.cert-manager.io -n istio-system -wNow deploy the sample application httpbin in the labeled namespace. Note the use of a
variable to match the manifest version to your installed istio version:
kubectl apply -n $NAMESPACE -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/httpbin/httpbin.yamlYou should see something similar to the output here for certificaterequests:
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
istio-ca-74bnl True True selfsigned system:serviceaccount:cert-manager:cert-manager 2d2h
istiod-w9zh6 True True istio-ca system:serviceaccount:cert-manager:cert-manager 27m
istio-csr-8ddcs istio-ca system:serviceaccount:cert-manager:cert-manager-istio-csr 0s
istio-csr-8ddcs True istio-ca system:serviceaccount:cert-manager:cert-manager-istio-csr 0s
istio-csr-8ddcs True True istio-ca system:serviceaccount:cert-manager:cert-manager-istio-csr 0s
istio-csr-8ddcs True True istio-ca system:serviceaccount:cert-manager:cert-manager-istio-csr 0s
The key request being istio-csr-8ddcs in our example output. You should then check your
cert-manager log output for two log lines with this request being "Approved" and "Ready":
I0113 16:51:59.186482 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "istio-csr-8ddcs" condition "Approved" to 2022-01-13 16:51:59.186455713 +0000 UTC m=+3507.098466775
I0113 16:51:59.258876 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "istio-csr-8ddcs" condition "Ready" to 2022-01-13 16:51:59.258837897 +0000 UTC m=+3507.170859959
You should now see the application is running with both the application container and the sidecar:
~ kubectl get pods -n $NAMESPACE
NAME READY STATUS RESTARTS AGE
httpbin-74fb669cc6-559cg 2/2 Running 0 4mTo validate that the istio-proxy sidecar container has requested the certifiate from the correct
service, check the container logs:
kubectl logs $(kubectl get pod -n $NAMESPACE -o jsonpath="{.items...metadata.name}" --selector app=$APP) -c istio-proxyYou should see some early logs similar to this example:
2022-01-13T16:51:58.495493Z info CA Endpoint cert-manager-istio-csr.cert-manager.svc:443, provider Citadel
2022-01-13T16:51:58.495817Z info Using CA cert-manager-istio-csr.cert-manager.svc:443 cert with certs: var/run/secrets/istio/root-cert.pem
2022-01-13T16:51:58.495941Z info citadelclient Citadel client using custom root cert: cert-manager-istio-csr.cert-manager.svc:443
Finally we can inspect the certificate being used in memory by Envoy. This one liner should return you the certificate being used:
istioctl proxy-config secret $(kubectl get pods -n $NAMESPACE -o jsonpath='{.items..metadata.name}' --selector app=$APP) -o json | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 --decode | openssl x509 -text -nooutIn particular look for the following sections:
Signature Algorithm: ecdsa-with-SHA256
Issuer: O=cert-manager, O=cluster.local, CN=istio-ca
Validity
Not Before: Jan 13 16:51:59 2022 GMT
Not After : Jan 13 17:51:59 2022 GMT
...
X509v3 Subject Alternative Name:
URI:spiffe://cluster.local/ns/default/sa/httpbin
You should see the relevant Trust Domain inside the Issuer. In the default case, it should be:
cluster.local as above. Note that the SPIFFE URI may be different if you used a different
namespace or application.
Assuming your running inside kind, you can simply remove the cluster:
kind delete cluster kind