post-master-upgrade fails on 'Waiting for redis instance to be removed'

[adamwalach]

Description
post-master-kyma-gke-upgrade fails randomly on "Test Kyma end-to-end upgrade scenarios" stage

Example: https://storage.googleapis.com/kyma-prow-logs/logs/post-master-kyma-gke-upgrade/1185157315163066369/build-log.txt

Actual result

{"level":"info","log":{"message":"Waiting for redis instance to be removed","taskID":"8cd2c541-6fb4-4a03-8546-3db703c2ac05","time":"2019-10-18T12:24:44.529Z"}}
{"level":"error","log":{"message":"timed out waiting for the condition","taskID":"8cd2c541-6fb4-4a03-8546-3db703c2ac05","time":"2019-10-18T12:26:44.538Z"}}

Cluster State:
fail-log.log

• ServiceInstance status: All associated ServiceBindings must be removed before this ServiceInstance can be deleted
• ServiceBinding status: Unbind request for ServiceBinding in-flight to Broker

Migration log:
migration-log.log

Steps to reproduce

Fails randomly

[jasiu001]

As mentioned in the issue description, the error occurs because ServiceBinding cannot be deleted.
ServiceBinding during unbind (remove) process is set to "in-flight" status, the process never ends so SB stays in that status.
ServiceCatalog controller (during unbind SB process) tries to find parent ServiceInstance, after finding it checks its ServiceClass reference.
Logs from ServiceCatalog controller:

I 2019-10-18T12:24:44.496966Z ServiceBinding "helmbrokerupgradetest/redis-credentials" v19161: Processing Delete
(...)
I 2019-10-18T12:24:44.655132Z Error syncing ServiceBinding helmbrokerupgradetest/redis-credentials (retry: 0/15): ClusterServiceClass reference for Instance has not been resolved yet
(...)
I 2019-10-18T12:26:07.070575Z Error syncing ServiceBinding helmbrokerupgradetest/redis-credentials (retry: 14/15): ClusterServiceClass reference for Instance has not been resolved yet

the code responsible for this check is located in controller binding

When we look at ServiceInstance spec then we see lack of reference class:

"spec": {
        "clusterServiceClassExternalName": "redis",
        "clusterServicePlanExternalName": "micro",
        "externalID": "f944b7ee-f19e-11e9-b995-be76fb442d49",
        "userInfo": {
            (...)
        },
        "updateRequests": 0
    },

This lack of reference comes from the migration process, more specifically from the restore ServiceInstance process.
Logs from migration jobs shows that the ServiceInstance restore process failed in the first iteration:

I 2019-10-18T12:11:54.589743Z Processing Service Instance: api-service-id
I 2019-10-18T12:11:55.181101Z Processing Service Instance: events-service-id
I 2019-10-18T12:11:55.782833Z Processing Service Instance: redis
I 2019-10-18T12:11:56.218256Z Retry 2
I 2019-10-18T12:11:56.218300Z Processing Service Instance: redis
I 2019-10-18T12:11:56.388871Z Resource already exists, deleting and recreating
I 2019-10-18T12:11:57.380901Z Applying 2 service bindings
I 2019-10-18T12:11:57.380944Z Removing owner referneces from secrets

The bug is described and fixed in this PR