Create the migration process between api-server and CRDs in Service Catalog

[mszostok]

Description

We need to create a migration guide/scripts which will allow clients to do easily migraton from api-server to CRD approach.

AC

• migration process is well documented (3MD)
• upgrade with migration only if previous SC version has api-server (1MD)
• check SBU in case SB is deleted. We should keep them (2MD)
• block API-Server mutations operation before backup (probably by webhook functionality) (2MD)
• SPIKE: for "Helm rollback to previous release" from CRD to api-server (hook:rollback) (1MD)
• update CRD chart with migration-tool (1MD)
• provide script for which is doing more or less what our Kyma upgrade plan is doing. Designed to Service catalog upstream. (3MD)

[PK85]

Issues to be created:

• find critical points that we need to create a contracts
• more or less mock critical paths and execute upgrade with all steps
• execute test with our upgrade plan per PR
• provide a way with dryRun

[mszostok]

remember that our solution should work both with dedicated etcd (installed by ServiceCatalog chart) and also with scenario where user used already existing etcd.

For example, we cannot have assumption that we are not triggering the deletion of the CustomResource because we will always delete the underlying storage. In some cases that could not be true and in restore phase we can have some conflict.

[piotrmiskiewicz]

PoC: manual steps which performs the migration: https://github.com/piotrmiskiewicz/service-catalog/blob/to-crd-migration/contrib/hack/crd/migration.md
manual upgrade scenario:
Install Minikue, Tiller and Service Catalog:

cd contrib/hack/crd

set -u
set -o errexit

CURRENT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

echo "- Initialize Minikube"
bash ${CURRENT_DIR}/minikube.sh

echo "- Installing Tiller..."
kubectl apply -f ${CURRENT_DIR}/../assets/tiller.yaml

bash ${CURRENT_DIR}/is-ready.sh kube-system name tiller

echo "- Installing SC with API Server"
helm repo add svc-cat https://svc-catalog-charts.storage.googleapis.com
helm install svc-cat/catalog \
    --name catalog --namespace catalog

Create SC resources

Perform backup:

kubectl get serviceinstance -o yaml --export --all-namespaces > serviceinstance.yaml
kubectl get servicebinding -o yaml --export --all-namespaces > servicebinding.yaml
kubectl get clusterserviceclass -o yaml --export --all-namespaces > clusterserviceclass.yaml
kubectl get clusterserviceplan -o yaml --export --all-namespaces > clusterserviceplan.yaml
kubectl get clusterservicebroker -o yaml --export --all-namespaces > clusterservicebroker.yaml
kubectl get serviceclass -o yaml --export --all-namespaces > serviceclass.yaml
kubectl get serviceplan -o yaml --export --all-namespaces > serviceplan.yaml
kubectl get servicebroker -o yaml --export --all-namespaces > servicebroker.yaml

Delete existing SC:

helm delete catalog --purge

Install new SC (with CRDs)

helm install ../../../charts/catalog --name catalog --namespace kyma-system

Scale down the controller manager

kubectl -n kyma-system scale deploy --replicas=0 catalog-catalog-controller-manager

Import resources

kubectl apply -f serviceinstance.yaml
kubectl apply -f servicebinding.yaml
kubectl apply -f clusterserviceclass.yaml
kubectl apply -f clusterserviceplan.yaml
kubectl apply -f clusterservicebroker.yaml
kubectl apply -f serviceclass.yaml
kubectl apply -f serviceplan.yaml
kubectl apply -f servicebroker.yaml

scale up:

kubectl -n kyma-system scale deploy --replicas=1 catalog-catalog-controller-manager

[piotrmiskiewicz]

TODOs:

• make Service Catalog CRDs version chart upgradeable, AC: the upgrade must work fine helm upgrade catalog <path to chart> (MS) ✅
• script which installs the Service-Catalog with API Server, test-broker, creates sample resources (namespaced and cluster wide) then performs an upgrade SC, then test resources (PJ) in progress
• test and prepare SC configuration which uses main k8s ETCD (PM)✅
• documentation - in progress
• backup/upgrade tool which stores existing SC resources and applies after new SC starts (it should also scale down/up controller manager). In case of API Server use etcd - resources must be removed from etcd (PM) in progress

[piotrmiskiewicz]

Using main ETCD in Minikube:

./bin/minikube.sh

copy certs and create a proper secret:

kubectl cp --namespace kube-system kube-apiserver-minikube:var/lib/minikube/certs/apiserver-etcd-client.crt etcd-client.crt
kubectl cp --namespace kube-system kube-apiserver-minikube:var/lib/minikube/certs/apiserver-etcd-client.key etcd-client.key
kubectl cp --namespace kube-system kube-apiserver-minikube:var/lib/minikube/certs/etcd/ca.crt etcd-client-ca.crt

kubectl create ns kyma-system

kubectl create secret generic -n kyma-system service-catalog-apiserver-cert --from-file=etcd-client.key --from-file=etcd-client.crt --from-file=etcd-client-ca.crt

install SC:

helm repo add svc-cat https://svc-catalog-charts.storage.googleapis.com
helm install svc-cat/catalog --set apiserver.storage.etcd.servers=https://127.0.0.1:2379 --set apiserver.storage.etcd.tls.enabled=true --set apiserver.storage.etcd.tls.clientCertSecretName=service-catalog-apiserver-cert --set apiserver.storage.etcd.useEmbedded=false \
    --name catalog --namespace catalog

create resources, then connect directly to etcd and get it, for example:

kubectl port-forward -n kube-system etcd-minikube 2379:2379 &
ETCDCTL_API=3 etcdctl --cacert=etcd-client-ca.crt --key=etcd-client.key --cert=etcd-client.crt get /registry/servicecatalog.k8s.io/clusterservicebrokers/helm-broker --endpoints=https://127.0.0.1:2379

gives an output:

/registry/servicecatalog.k8s.io/clusterservicebrokers/helm-broker
{"kind":"ClusterServiceBroker","apiVersion":"servicecatalog.k8s.io/v1beta1","metadata":{"name":"helm-broker","uid":"07aa039e-869d-11e9-8d41-9242bee4c30c","generation":1,"creationTimestamp":"2019-06-04T07:47:40Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"servicecatalog.k8s.io/v1beta1\",\"kind\":\"ClusterServiceBroker\",\"metadata\":{\"annotations\":{},\"name\":\"helm-broker\"},\"spec\":{\"relistRequests\":1,\"url\":\"http://helm-broker/\"}}\n"},"finalizers":["kubernetes-incubator/service-catalog"]},"spec":{"url":"http://helm-broker/","relistBehavior":"Duration","relistRequests":1},"status":{"conditions":[{"type":"Ready","status":"False","lastTransitionTime":"2019-06-04T07:47:41Z","reason":"ErrorFetchingCatalog","message":"Error fetching catalog.Error getting broker catalog: Get http://helm-broker/v2/catalog: dial tcp: lookup helm-broker on 10.96.0.10:53: no such host"}],"reconciledGeneration":0,"operationStartTime":"2019-06-04T07:47:41Z"}}

Because all resource are stored in the same place in ETCD as CDR version does, we cannot apply raw data which were stored in a file (backup). Resources should be applied without "metadata.resourceVersion"

after installing ServiceCatalog we can see object in the ETCD:

{"apiVersion":"servicecatalog.k8s.io/v1beta1","kind":"ClusterServiceBroker","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"servicecatalog.k8s.io/v1beta1\",\"kind\":\"ClusterServiceBroker\",\"metadata\":{\"annotations\":{},\"creationTimestamp\":\"2019-06-04T09:15:48Z\",\"finalizers\":[\"kubernetes-incubator/service-catalog\"],\"generation\":1,\"name\":\"helm-broker\"},\"spec\":{\"relistBehavior\":\"Duration\",\"relistRequests\":1,\"url\":\"http://helm-broker/\"},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2019-06-04T09:15:49Z\",\"message\":\"Error fetching catalog.Error getting broker catalog: Get http://helm-broker/v2/catalog: dial tcp: lookup helm-broker on 10.96.0.10:53: no such host\",\"reason\":\"ErrorFetchingCatalog\",\"status\":\"False\",\"type\":\"Ready\"}],\"operationStartTime\":\"2019-06-04T09:15:49Z\",\"reconciledGeneration\":0}}\n"},"creationTimestamp":"2019-06-04T09:15:48Z","finalizers":["kubernetes-incubator/service-catalog"],"generation":1,"name":"helm-broker","resourceVersion":"","selfLink":"","uid":"579dd5c9-86a9-11e9-984c-9242bee4c30c"},"spec":{"relistBehavior":"Duration","relistRequests":1,"url":"http://helm-broker/"},"status":{"conditions":[{"lastTransitionTime":"2019-06-04T09:15:49Z","message":"Error fetching catalog.Error getting broker catalog: Get http://helm-broker/v2/catalog: dial tcp: lookup helm-broker on 10.96.0.10:53: no such host","reason":"ErrorFetchingCatalog","status":"False","type":"Ready"}],"lastConditionState":"ErrorFetchingCatalog","operationStartTime":"2019-06-04T09:15:49Z","reconciledGeneration":0}}```
the `"lastConditionState":"ErrorFetchingCatalog` was added.

[mszostok]

Make Service Catalog CRDs version chart upgradeable

Problems encountered when executing helm upgrade against the Service Catalog release with Aggregated API Server

• Port already allocated

  UPGRADE FAILED
  Error: failed to create resource: Service "catalog-catalog-webhook" is invalid: 
  spec.ports[0].nodePort: Invalid value: 30443: provided port is already allocated
  Error: UPGRADE FAILED: failed to create resource: Service "catalog-catalog-webhook" is invalid: spec.ports[0].nodePort: Invalid value: 30443: provided port is already allocated
  

  this was solved by changing the webhook port from 30443 to 31443. Commit: kubernetes-retired/service-catalog@e5d27e2

• not found error

  Error: UPGRADE FAILED: no Secret with the name "catalog-catalog-webhook-cert" found
  

  This issue was already reported in Helm and fixed in version 2.14 by introducing the flag -cleanup-on-fail.

  This problem occurs only if the service catalog release is marked as failed because of the problem from the first point.

[PK85]

Initial Raw scenario:

• pre-upgrade hook
  • replace the api-server deployment with the api-server that has a read-only mode
  • backup the Service Catalog resources to the persistent volume (consider backuping etcd and second with new format that will be used for restore)
• upgrade
  • remove the api-server
  • remove the etcd storage
  • adjust secrets, RBAC etc.
  • upgrade controller manager
  • install webhook server
• post-upgrade
  • scale down the controller manager to 0
  • restore the Service Catalog resources - Spec and Status (status is important because we do not want to trigger provisioning for already process items)
  • scale up controller manager