Automated Service Deployment

Using Keel and Harbor for automated image deployment

Keel is an operator for tracking image updates and updating Kubernetes resources accordingly. Apart from polling and webhooks notifications for deployment updates (Deployments, StatefulSet, DaemonSet, Helm), it features an approval system through a Slack bot and update notifications integrated with some chat platforms.

It is a much simpler approach than using a full-fledged GitOps solution like ArgoCD or FluxCD. We think the full-blow GitOps approach is overkill for our use case and might contribute negatively to the accumulated technical debt.

Kind development cluster

We assume you followed the kind cluster config in the Development Cluster and if so the service IP's will be in the 172.28.0.0/16 subnet.

We assume that Harbor will take the first IP in the range 172.18.255.200 and that Keel will take the second 172.18.255.201, following the order of configuration in the tutorial. If you're using another subnetwork, pay attention to places where this might be problematic, namely the cluster config setting for configuring the TLS ignore option for containerd instances running in each kind node (config file used in the kind cluster config instructions). This option allows images from unverified image registries, like the Harbor instance to be configured in the kind development cluster, to be used by the cluster. This option is not necessary if the image registry has a valid TLS certificate.

In order to enable the cluster to use images from unverified registries, you should also add the IP of the Harbor service to the docker config's insecure registries option on your machine. You should have some config similar to this in /etc/docker/daemon.json:

{
  "log-level": "warn",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  },
  "insecure-registries" : [
    "172.28.255.200"
   ]
}

Be sure to restart the docker deamon after changing the config.

sudo systemctl restart docker

Deploy the Harbor image registry and Keel

Prerequisites

• Helm

The following script installs Harbor using Helm charts and Keel with a Kubernetes manifest.

./services/image-registry/deploy-dev.sh

Harbor credentials

• username: admin
• password: Harbor12345

Keel credentials

• username: username
• password: password

Create a deployment with Keel annotations for deployment automation

Note

Make sure to tag and push your images to Harbor before applying the deployment.

docker pull alpine:3.16.3
docker login -u admin -p Harbor12345 172.28.255.200

# Generate multiple tags to test
docker tag alpine:3.16.3 172.28.255.200/library/alpine:3.16.3
docker tag alpine:3.16.3 172.28.255.200/library/alpine:3.16.4
docker tag alpine:3.16.3 172.28.255.200/library/alpine:3.16.5

docker push 172.28.255.200/library/alpine:3.16.3

Check the provided example-deployment. More configurations are available in the Keel documentation.

kubectl apply -f services/image-registry/example-deployment.yaml

Add a webhook in Harbor

Harbor's interface is available at:

Harbor dashboard: http://172.28.255.200/
Credentials above.

In Harbor, add a webhook for artifact push events in the project with your image. Use the Fully Qualified Domain Name (FQDN) of the keel service and the correct endpoint for Harbor events. FQDNs follow the following pattern in Kubernetes <service-name>.<namespace>.svc.<cluster-domain>; usually, the cluster domain is cluster.local. It should look something like this:

http://keel.keel.svc.cluster.local:9300/v1/webhooks/harbor

Watch it all from the Keel dashboard

The Keel dashboard shows all the images it tracks and allows some on-the-fly modifications to update policies.

Keel dashboard: http://172.28.255.201:9300/
Credentials above.

Run the following and watch the container version in the deployment change with it. You can do it through the dashboard or using kubectl.

docker push 172.28.255.200/library/alpine:3.16.4 
kubectl describe deployment alpine # Check the image tag

# You can do it again with the 3.16.5 tag

Using the in-house GitHub action to push images directly from GitHub to Harbor

If you already have a cluster running, you can use the GitHub Action developed specifically for this purpose, which can be found at NIAEFEUP/push-to-niployments.

Instructions on how to use this can be found in the repository's README.md file.

A sample project using this action can be found here

Warning

Currently this action only works for Image registries served behind HTTPS, due to DOcker Buildx pushing images using HTTPS by default. This can be configured, however the first version of the GH Action will not support it.