Dataplanes cannot connect to a non-default Mesh with mTLS enabled on k8s

[yskopets]

Summary

If a user creates a non-default Mesh with mTLS enabled on k8s, dataplanes cannot connect to it.

Steps To Reproduce

1. Create a new Mesh with mTLS enabled, e.g.

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  namespace: kuma-system
  name: pilot
spec:
  mtls:
    enabled: true
    ca:
      builtin: {}

2. Deploy a sample app into that Mesh, e.g.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kuma-tcp-echo
  namespace: kuma-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: kuma-tcp-echo
  template:
    metadata:
      labels:
        app: kuma-tcp-echo
      annotations:
        kuma.io/mesh: pilot
    spec:
      containers:
      - name: kuma-tcp-echo
        image: kong-docker-kuma-docker.bintray.io/kuma-tcp-echo:0.1.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8000

notice kuma.io/mesh: pilot annotation

3. All requests through that dataplane are failing and the log is filled with warnings

[2019-09-18 09:54:57.503][22][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to retrive Root Certificates of a given Builtin CA: failed to load CA key pair for Mesh "pilot": Resource not found: type="Secret" namespace="kuma-system" name="builtinca.pilot" mesh=""
[2019-09-18 09:55:02.442][22][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to generate a Workload Identity Certificate for {Mesh:pilot Service:demo-app.kuma-demo.svc:80}: failed to load CA key pair for Mesh "pilot": Resource not found: type="Secret" namespace="kuma-system" name="builtinca.pilot" mesh=""

Additional Details & Logs

• Version: 0.1.2
• Error logs: see above
• Configuration: kumactl install control-plane
• Platform and Operating System: k8s