fix master e2e tests #1163

ryandawsonuk · 2019-11-29T20:42:20Z

This is to fix #1164

First need to give operator read access on configmaps as found in #1162

ryandawsonuk · 2019-11-29T20:42:54Z

/test integration

ryandawsonuk · 2019-11-29T20:53:14Z

ok now I'm seeing different problems in the operator logs during the e2e tests:

2019-11-29T20:50:17.182Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"ConfigMap","namespace":"seldon-system","name":"controller-leader-election-helper","uid":"7741775c-73e2-44ad-8a1d-b2d7e6887c55","apiVersion":"v1","resourceVersion":"1555"}, "reason": "LeaderElection", "message": "seldon-controller-manager-66b99785b8-26pqv_cf1f4862-12e9-11ea-90f5-5a78af63fead became leader"}
2019-11-29T20:50:17.283Z        INFO    controller-runtime.controller   Starting workers        {"controller": "seldondeployment", "worker count": 1}
2019-11-29T20:50:17.283Z        INFO    controllers.SeldonDeployment    pSvcName        {"seldondeployment": "test-single-model/mymodel", "val": "mymodel-mymodel-mymodel"}
2019-11-29T20:50:17.584Z        ERROR   seldondeployment        No entry in predictors map for UNKNOWN_IMPLEMENTATION
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
github.com/seldonio/seldon-core/operator/api/v1alpha2.GetPrepackServerConfig
        /workspace/api/v1alpha2/seldondeployment_webhook.go:136
github.com/seldonio/seldon-core/operator/controllers.createStandaloneModelServers
        /workspace/controllers/seldondeployment_prepackaged_servers.go:279
github.com/seldonio/seldon-core/operator/controllers.createComponents
        /workspace/controllers/seldondeployment_controller.go:444
github.com/seldonio/seldon-core/operator/controllers.(*SeldonDeploymentReconciler).Reconcile
        /workspace/controllers/seldondeployment_controller.go:1221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
2019-11-29T20:50:17.586Z        INFO    controllers.SeldonDeployment    Scheme  {"seldondeployment": "test-single-model/mymodel", "r.scheme": {}}
2019-11-29T20:50:17.586Z        INFO    controllers.SeldonDeployment    createDeployments       {"seldondeployment": "test-single-model/mymodel", "deploy": {"namespace": "test-single-model", "name": "mymodel-mymodel-7cd068f"}}
2019-11-29T20:50:17.586Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"seldondeployment": "test-single-model/mymodel", "namespace": "test-single-model", "name": "mymodel-mymodel-7cd068f", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"mymodel-mymodel-7cd068f-6d8bcdcc5\" is progressing."}]}}
2019-11-29T20:50:17.586Z        INFO    controllers.SeldonDeployment    Deployment status       {"seldondeployment": "test-single-model/mymodel", "name": "mymodel-mymodel-7cd068f", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"mymodel-mymodel-7cd068f-6d8bcdcc5\" is progressing."}]}}
2019-11-29T20:50:17.587Z        INFO    controllers.SeldonDeployment    Found identical Service {"seldondeployment": "test-single-model/mymodel", "namespace": "test-single-model", "name": "mymodel-mymodel-classifier-seldonio-mock-classifier-1-0", "status": {"loadBalancer":{}}}
2019-11-29T20:50:17.587Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"seldondeployment": "test-single-model/mymodel"}
2019-11-29T20:50:17.783Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "test-single-model/mymodel"}
2019-11-29T20:50:17.783Z        INFO    controllers.SeldonDeployment    pSvcName        {"seldondeployment": "test-abtest-model/myabtest", "val": "myabtest-myabtest-myabtest"}
2019-11-29T20:50:17.783Z        ERROR   seldondeployment        No entry in predictors map for RANDOM_ABTEST
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
github.com/seldonio/seldon-core/operator/api/v1alpha2.GetPrepackServerConfig
        /workspace/api/v1alpha2/seldondeployment_webhook.go:136
github.com/seldonio/seldon-core/operator/controllers.createStandaloneModelServers
        /workspace/controllers/seldondeployment_prepackaged_servers.go:279
github.com/seldonio/seldon-core/operator/controllers.createComponents
        /workspace/controllers/seldondeployment_controller.go:444
github.com/seldonio/seldon-core/operator/controllers.(*SeldonDeploymentReconciler).Reconcile
        /workspace/controllers/seldondeployment_controller.go:1221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
2019-11-29T20:50:17.784Z        ERROR   seldondeployment        No entry in predictors map for RANDOM_ABTEST
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
github.com/seldonio/seldon-core/operator/api/v1alpha2.GetPrepackServerConfig
        /workspace/api/v1alpha2/seldondeployment_webhook.go:136
github.com/seldonio/seldon-core/operator/api/v1alpha2.SetImageNameForPrepackContainer
        /workspace/api/v1alpha2/seldondeployment_webhook.go:146
github.com/seldonio/seldon-core/operator/controllers.addModelDefaultServers
        /workspace/controllers/seldondeployment_prepackaged_servers.go:141
github.com/seldonio/seldon-core/operator/controllers.createStandaloneModelServers
        /workspace/controllers/seldondeployment_prepackaged_servers.go:281
github.com/seldonio/seldon-core/operator/controllers.createComponents
        /workspace/controllers/seldondeployment_controller.go:444
github.com/seldonio/seldon-core/operator/controllers.(*SeldonDeploymentReconciler).Reconcile
        /workspace/controllers/seldondeployment_controller.go:1221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
2019-11-29T20:50:17.784Z        ERROR   seldondeployment        No entry in predictors map for UNKNOWN_IMPLEMENTATION
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
github.com/seldonio/seldon-core/operator/api/v1alpha2.GetPrepackServerConfig
        /workspace/api/v1alpha2/seldondeployment_webhook.go:136
github.com/seldonio/seldon-core/operator/controllers.createStandaloneModelServers
        /workspace/controllers/seldondeployment_prepackaged_servers.go:279
github.com/seldonio/seldon-core/operator/controllers.createStandaloneModelServers
        /workspace/controllers/seldondeployment_prepackaged_servers.go:307
github.com/seldonio/seldon-core/operator/controllers.createComponents
        /workspace/controllers/seldondeployment_controller.go:444
github.com/seldonio/seldon-core/operator/controllers.(*SeldonDeploymentReconciler).Reconcile
        /workspace/controllers/seldondeployment_controller.go:1221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
2019-11-29T20:50:17.784Z        ERROR   seldondeployment        No entry in predictors map for UNKNOWN_IMPLEMENTATION
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
github.com/seldonio/seldon-core/operator/api/v1alpha2.GetPrepackServerConfig
        /workspace/api/v1alpha2/seldondeployment_webhook.go:136
github.com/seldonio/seldon-core/operator/controllers.createStandaloneModelServers
        /workspace/controllers/seldondeployment_prepackaged_servers.go:279
github.com/seldonio/seldon-core/operator/controllers.createStandaloneModelServers
        /workspace/controllers/seldondeployment_prepackaged_servers.go:307
github.com/seldonio/seldon-core/operator/controllers.createComponents
        /workspace/controllers/seldondeployment_controller.go:444
github.com/seldonio/seldon-core/operator/controllers.(*SeldonDeploymentReconciler).Reconcile
        /workspace/controllers/seldondeployment_controller.go:1221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    Scheme  {"seldondeployment": "test-abtest-model/myabtest", "r.scheme": {}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    createDeployments       {"seldondeployment": "test-abtest-model/myabtest", "deploy": {"namespace": "test-abtest-model", "name": "myabtest-myabtest-41de5b8"}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"seldondeployment": "test-abtest-model/myabtest", "namespace": "test-abtest-model", "name": "myabtest-myabtest-41de5b8", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"myabtest-myabtest-41de5b8-7c676c959b\" is progressing."}]}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    Deployment status       {"seldondeployment": "test-abtest-model/myabtest", "name": "myabtest-myabtest-41de5b8", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"myabtest-myabtest-41de5b8-7c676c959b\" is progressing."}]}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    Scheme  {"seldondeployment": "test-abtest-model/myabtest", "r.scheme": {}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    createDeployments       {"seldondeployment": "test-abtest-model/myabtest", "deploy": {"namespace": "test-abtest-model", "name": "myabtest-myabtest-df66c5c"}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"seldondeployment": "test-abtest-model/myabtest", "namespace": "test-abtest-model", "name": "myabtest-myabtest-df66c5c", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"myabtest-myabtest-df66c5c-688fc59657\" is progressing."}]}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    Deployment status       {"seldondeployment": "test-abtest-model/myabtest", "name": "myabtest-myabtest-df66c5c", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"myabtest-myabtest-df66c5c-688fc59657\" is progressing."}]}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    Scheme  {"seldondeployment": "test-abtest-model/myabtest", "r.scheme": {}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    createDeployments       {"seldondeployment": "test-abtest-model/myabtest", "deploy": {"namespace": "test-abtest-model", "name": "myabtest-myabtest"}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"seldondeployment": "test-abtest-model/myabtest", "namespace": "test-abtest-model", "name": "myabtest-myabtest", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"myabtest-myabtest-6c75fc4db6\" is progressing."}]}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    Deployment status       {"seldondeployment": "test-abtest-model/myabtest", "name": "myabtest-myabtest", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2019-11-29T20:49:25Z","lastTransitionTime":"2019-11-29T20:49:25Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"myabtest-myabtest-6c75fc4db6\" is progressing."}]}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    Found identical Service {"seldondeployment": "test-abtest-model/myabtest", "namespace": "test-abtest-model", "name": "myabtest-myabtest-classifier-1-seldonio-mock-classifier-1-0", "status": {"loadBalancer":{}}}
2019-11-29T20:50:17.784Z        INFO    controllers.SeldonDeployment    Found identical Service {"seldondeployment": "test-abtest-model/myabtest", "namespace": "test-abtest-model", "name": "myabtest-myabtest-classifier-2-seldonio-mock-classifier-1-0", "status": {"loadBalancer":{}}}
E1129 20:50:17.784975       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:76
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:65
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/workspace/controllers/seldondeployment_controller.go:885
/workspace/controllers/seldondeployment_controller.go:1231
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1337
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb8 pc=0x1184899]

goroutine 274 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:58 +0x105
panic(0x12e0460, 0x2165920)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/seldonio/seldon-core/operator/controllers.createServices(0xc0003a3b30, 0xc00068e500, 0xc0006ac9c0, 0x165e400, 0x165e420, 0xc000303420, 0x0, 0x0, 0xc0006ac9c0)
        /workspace/controllers/seldondeployment_controller.go:886 +0xa9
github.com/seldonio/seldon-core/operator/controllers.(*SeldonDeploymentReconciler).Reconcile(0xc0003a3b30, 0xc0008b1280, 0x11, 0xc000859d70, 0x8, 0x2179b80, 0x42bd21, 0x1632600, 0xc00029fd88)
        /workspace/controllers/seldondeployment_controller.go:1231 +0x397
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000f2140, 0x1329480, 0xc0008dc960, 0x1329400)
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216 +0x149
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000f2140, 0xc00036e000)
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192 +0xb5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc0000f2140)
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0003c6780)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x54
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0003c6780, 0x3b9aca00, 0x0, 0xc000244001, 0xc0000aa180)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc0003c6780, 0x3b9aca00, 0xc0000aa180)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:157 +0x311

This seems to be a bug that I've introduced.

ryandawsonuk · 2019-11-30T11:18:07Z

/test integration

ryandawsonuk · 2019-11-30T11:35:27Z

There was a problem with impl entries such as UNKNOWN_IMPLEMENTATION and RANDOM_ABTEST getting treated as prepack.

(Need to follow-up on how the operator image is set for local e2e tests. I'm seeing clear discrepancies between image in e2e tests and the code in my workspace. I actually had to push the image before the e2e tests picked up the changes.)

Think that is gone now as https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1163/6.log#L3511 only had one failure and that looks like the intermittent ambassador one.

ryandawsonuk · 2019-11-30T11:50:32Z

/test integration

ryandawsonuk · 2019-11-30T12:06:35Z

e2e tests now passing - https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1163/7.log#L4508

ryandawsonuk · 2019-11-30T12:16:08Z

/ok-to-test
/lgtm
/approve

seldondev · 2019-11-30T12:16:09Z

@ryandawsonuk: you cannot LGTM your own PR.

In response to this:

/ok-to-test
/lgtm
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

seldondev · 2019-11-30T12:16:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ryandawsonuk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ryandawsonuk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

seldondev · 2019-11-30T12:46:15Z

@ryandawsonuk: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
integration	`af6cd31`	link	`/test integration`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ryandawsonuk · 2019-11-30T12:48:21Z

Merging now. The only 'failure' is that the docker process in the job didn't stop so a test infra problem, not a code issue.

grant read on configmaps

897f76f

seldondev added the size/S label Nov 29, 2019

ryandawsonuk mentioned this pull request Nov 29, 2019

e2e tests broken in master #1164

Closed

ryandawsonuk changed the title ~~grant read on configmaps~~ fix master e2e tests Nov 29, 2019

correction to IsPrepack check

90a66c3

seldondev added size/M and removed size/S labels Nov 30, 2019

make easier to debug

af6cd31

clarify run locally step

54de1b2

seldondev added the ok-to-test label Nov 30, 2019

seldondev added the approved label Nov 30, 2019

ryandawsonuk merged commit f9972d3 into SeldonIO:master Nov 30, 2019

ryandawsonuk mentioned this pull request Dec 2, 2019

local e2e don't build operator image #1171

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix master e2e tests #1163

fix master e2e tests #1163

ryandawsonuk commented Nov 29, 2019 •

edited

Loading

ryandawsonuk commented Nov 29, 2019

ryandawsonuk commented Nov 29, 2019

ryandawsonuk commented Nov 30, 2019

ryandawsonuk commented Nov 30, 2019 •

edited

Loading

ryandawsonuk commented Nov 30, 2019

ryandawsonuk commented Nov 30, 2019

ryandawsonuk commented Nov 30, 2019

seldondev commented Nov 30, 2019

seldondev commented Nov 30, 2019

seldondev commented Nov 30, 2019

ryandawsonuk commented Nov 30, 2019

fix master e2e tests #1163

fix master e2e tests #1163

Conversation

ryandawsonuk commented Nov 29, 2019 • edited Loading

ryandawsonuk commented Nov 29, 2019

ryandawsonuk commented Nov 29, 2019

ryandawsonuk commented Nov 30, 2019

ryandawsonuk commented Nov 30, 2019 • edited Loading

ryandawsonuk commented Nov 30, 2019

ryandawsonuk commented Nov 30, 2019

ryandawsonuk commented Nov 30, 2019

seldondev commented Nov 30, 2019

seldondev commented Nov 30, 2019

seldondev commented Nov 30, 2019

ryandawsonuk commented Nov 30, 2019

ryandawsonuk commented Nov 29, 2019 •

edited

Loading

ryandawsonuk commented Nov 30, 2019 •

edited

Loading