InfraTest sporadic failure on CI

[k-wall]

The io.enmasse.systemtest.standard.infra.InfraTest are seen to sporadically fail on Jenkins CI with a failure like this:

Error Message

Failure executing: POST at: https://10.0.133.19:8443/apis/enmasse.io/v1beta1/namespaces/enmasse-infra/addressspaces. Message: Unknown address space plan example-space-plan-standard. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=Unknown address space plan example-space-plan-standard, metadata=null, reason=Bad Request, status=Failure, additionalProperties={statusCode=400}).

Looking into the api-server log, I see deserialisation errors. These deserialisation errors explain why the api-server did not know about the newly created plan.

2019-07-15T01:19:23.682Z ERROR [WatchConnectionManager] Could not deserialize watch event: {"type":"ADDED","object":{"apiVersion":"admin.enmasse.io/v1beta1","kind":"StandardInfraConfig","metadata":{"creationTimestamp":"2019-07-15T01:19:23Z","generation":1,"name":"test-infra-1-standard","namespace":"enmasse-infra","resourceVersion":"55748","selfLink":"/apis/admin.enmasse.io/v1beta1/namespaces/enmasse-infra/standardinfraconfigs/test-infra-1-standard","uid":"946cdebb-a69e-11e9-a16a-fa163e2d2fe3"},"spec":{"admin":{"podTemplate":{"metadata":{"annotations":{},"labels":{"mycomponent":"admin"}},"spec":{"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"node-label-key","operator":"In","values":["myadminnode"]}]},"weight":1}]}},"containers":[],"nodeSelector":{},"tolerations":[{"effect":"NoSchedule","key":"admin","operator":"Exists"}]}},"resources":{"memory":"512Mi"}},"broker":{"addressFullPolicy":"FAIL","podTemplate":{"metadata":{"annotations":{},"labels":{"mycomponent":"broker"}},"spec":{"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"node-label-key","operator":"In","values":["mybrokernode"]}]},"weight":1}]}},"containers":[],"nodeSelector":{},"tolerations":[{"effect":"NoSchedule","key":"broker","operator":"Exists"}]}},"resources":{"memory":"512Mi","storage":"1Gi"}},"router":{"podTemplate":{"metadata":{"annotations":{},"labels":{"mycomponent":"router"}},"spec":{"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"node-label-key","operator":"In","values":["myrouternode"]}]},"weight":1}]}},"containers":[],"nodeSelector":{},"tolerations":[{"effect":"NoSchedule","key":"router","operator":"Exists"}]}},"resources":{"memory":"256Mi"}},"version":"0.29-SNAPSHOT"}}}
com.fasterxml.jackson.databind.JsonMappingException: No resource type found for:admin.enmasse.io/v1beta1#StandardInfraConfig
 at [Source: (String)"{"type":"ADDED","object":{"apiVersion":"admin.enmasse.io/v1beta1","kind":"StandardInfraConfig","metadata":{"creationTimestamp":"2019-07-15T01:19:23Z","generation":1,"name":"test-infra-1-standard","namespace":"enmasse-infra","resourceVersion":"55748","selfLink":"/apis/admin.enmasse.io/v1beta1/namespaces/enmasse-infra/standardinfraconfigs/test-infra-1-standard","uid":"946cdebb-a69e-11e9-a16a-fa163e2d2fe3"},"spec":{"admin":{"podTemplate":{"metadata":{"annotations":{},"labels":{"mycomponent":"admin""[truncated 1298 chars]; line: 1, column: 1796] (through reference chain: io.fabric8.kubernetes.api.model.WatchEvent["object"])
	at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:271)
	at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:1718)
	at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:78)
	at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:32)
	at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4013)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3004)
	at io.fabric8.kubernetes.client.dsl.internal.WatchHTTPManager.readWatchEvent(WatchHTTPManager.java:293)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:224)
	at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
	at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

2019-07-15T01:19:26.093Z ERROR [KubeSchemaApi] Error validating address space plan example-space-plan-standard, skipping
io.enmasse.k8s.api.SchemaValidationException: Error validating address space plan example-space-plan-standard: missing infra config definition test-infra-1-standard, found: [default, default-with-mqtt, default-minimal]
	at io.enmasse.k8s.api.KubeSchemaApi.validateAddressSpacePlan(KubeSchemaApi.java:107)
	at io.enmasse.k8s.api.KubeSchemaApi.assembleSchema(KubeSchemaApi.java:376)
	at io.enmasse.k8s.api.KubeSchemaApi.updateSchema(KubeSchemaApi.java:331)
	at io.enmasse.k8s.api.KubeSchemaApi.lambda$watchSchema$8(KubeSchemaApi.java:287)
	at io.enmasse.k8s.api.KubeCrdApi.lambda$watchResources$1(KubeCrdApi.java:72)
	at io.enmasse.k8s.api.cache.EventCache.pop(EventCache.java:82)
	at io.enmasse.k8s.api.cache.Reflector.run(Reflector.java:58)
	at io.enmasse.k8s.api.cache.Controller.lambda$start$0(Controller.java:22)
	at java.base/java.lang.Thread.run(Thread.java:834)

[k-wall]

EnMasses's registration of custom kinds looks sounds. I'm suspicious of the coding of io.fabric8.kubernetes.internal.KubernetesDeserializer. I think the io.fabric8.kubernetes.internal.KubernetesDeserializer#MAP ought to be synchronised or CHM as it is updated at runtime by many threads. I think this will explain the sporadic absence of the key from the map.

[k-wall]

Open issue/PR against kubernetes-client https://github.com/k-wall/kubernetes-client/pull/new/issue-1634-kubernetesdeserializer-threadsafety