Enabling bidirectional mounts on pod kills docker #4072

Madhu-1 · 2019-04-10T04:51:06Z

Enabling bidirectional mounts on pods kills docker.
deployed kubernetes using minikube with VirtualBox driver with (CPUs=4, Memory=5000MB, Disk=80000MB)

pod list output

[🎩︎]mrajanna@localhost rook $]kubectl get po -nsmoke-ns-system
NAME                                READY   STATUS              RESTARTS   AGE
csi-cephfsplugin-cg764              1/2     RunContainerError   0          85s
csi-cephfsplugin-provisioner-0      1/2     ErrImagePull        0          85s
csi-rbdplugin-attacher-0            1/1     Running             0          85s
csi-rbdplugin-provisioner-0         1/3     ImageInspectError   0          85s
csi-rbdplugin-vw4ch                 1/2     RunContainerError   0          85s
rook-ceph-agent-kqpzk               1/1     Running             0          85s
rook-ceph-operator-7f5858cb-ngrd5   1/1     Running             0          87s
rook-discover-dqdjh                 1/1     Running             0          85s

docker ps output

$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

kubectl describe output

  kubectl describe po/csi-rbdplugin-vw4ch -nsmoke-ns-system
Name:               csi-rbdplugin-vw4ch
Namespace:          smoke-ns-system
Priority:           0
PriorityClassName:  <none>
Node:               minikube/10.0.2.15
Start Time:         Wed, 10 Apr 2019 10:12:56 +0530
Labels:             app=csi-rbdplugin
                    controller-revision-hash=8469978b6d
                    pod-template-generation=1
Annotations:        <none>
Status:             Running
IP:                 10.0.2.15
Controlled By:      DaemonSet/csi-rbdplugin
Containers:
  driver-registrar:
    Container ID:  docker://b0c4a874b40af193951955f0673faf1b26b926d490f3500384d2d48922d2d6f3
    Image:         quay.io/k8scsi/csi-node-driver-registrar:v1.0.2
    Image ID:      docker-pullable://quay.io/k8scsi/csi-node-driver-registrar@sha256:ffecfbe6ae9f446e5102cbf2c73041d63ccf44bcfd72e2f2a62174a3a185eb69
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=/csi/csi.sock
      --kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock
    State:          Running
      Started:      Wed, 10 Apr 2019 10:13:03 +0530
    Ready:          True
    Restart Count:  0
    Environment:
      KUBE_NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /csi from plugin-dir (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-csi-rbd-plugin-sa-token-k2rnr (ro)
  csi-rbdplugin:
    Container ID:  docker://60d0529d7176e3256d142500e2de6120cb2ce60352127ba97baa6599d0b89a1c
    Image:         quay.io/cephcsi/rbdplugin:v1.0.0
    Image ID:      docker-pullable://quay.io/cephcsi/rbdplugin@sha256:893f9d27d3abfb3a4a263566f638af6b98c0660ad684f94ecd4e0586a90c05ec
    Port:          <none>
    Host Port:     <none>
    Args:
      --nodeid=$(NODE_ID)
      --endpoint=$(CSI_ENDPOINT)
      --v=5
      --drivername=rbd.csi.ceph.com
      --containerized=true
      --metadatastorage=k8s_configmap
    State:          Waiting
      Reason:       RunContainerError
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      OCI runtime create failed: open /var/run/docker/runtime-runc/moby/60d0529d7176e3256d142500e2de6120cb2ce60352127ba97baa6599d0b89a1c/state.json: no such file or directory: unknown
      Exit Code:    128
      Started:      Wed, 10 Apr 2019 10:13:53 +0530
      Finished:     Wed, 10 Apr 2019 10:13:53 +0530
    Ready:          False
    Restart Count:  0
    Environment:
      HOST_ROOTFS:    /rootfs
      NODE_ID:         (v1:spec.nodeName)
      POD_NAMESPACE:  smoke-ns-system (v1:metadata.namespace)
      CSI_ENDPOINT:   unix:///csi/csi.sock
    Mounts:
      /csi from plugin-dir (rw)
      /dev from host-dev (rw)
      /lib/modules from lib-modules (ro)
      /rootfs from host-rootfs (rw)
      /sys from host-sys (rw)
      /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/ from plugin-mount-dir (rw)
      /var/lib/kubelet/pods from pods-mount-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-csi-rbd-plugin-sa-token-k2rnr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  plugin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/rbd.csi.ceph.com
    HostPathType:  DirectoryOrCreate
  plugin-mount-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/
    HostPathType:  DirectoryOrCreate
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  Directory
  pods-mount-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:  Directory
  host-dev:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  
  host-rootfs:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  
  host-sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:  
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  rook-csi-rbd-plugin-sa-token-k2rnr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rook-csi-rbd-plugin-sa-token-k2rnr
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  2m1s  default-scheduler  Successfully assigned smoke-ns-system/csi-rbdplugin-vw4ch to minikube
  Normal   Pulling    2m    kubelet, minikube  pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
  Normal   Pulled     114s  kubelet, minikube  Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
  Normal   Created    114s  kubelet, minikube  Created container
  Normal   Started    114s  kubelet, minikube  Started container
  Normal   Pulling    114s  kubelet, minikube  pulling image "quay.io/cephcsi/rbdplugin:v1.0.0"
  Normal   Pulled     64s   kubelet, minikube  Successfully pulled image "quay.io/cephcsi/rbdplugin:v1.0.0"
  Normal   Created    64s   kubelet, minikube  Created container
  Warning  Failed     64s   kubelet, minikube  Error: failed to start container "csi-rbdplugin": Error response from daemon: OCI runtime create failed: open /var/run/docker/runtime-runc/moby/csi-rbdplugin/state.json: no such file or directory: unknown

kubelet service status

 kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Wed 2019-04-10 04:39:41 UTC; 7min ago
     Docs: http://kubernetes.io/docs/
 Main PID: 3381 (kubelet)
    Tasks: 22 (limit: 4915)
   CGroup: /system.slice/kubelet.service
           └─3381 /usr/bin/kubelet --allow-privileged=true --authorization-mode=Webhook --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --cgroup-driver=cgroupfs --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --container-runtime=docker --fail-swap-on=false --hostname-override=minikube --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests

Apr 10 04:46:53 minikube kubelet[3381]: E0410 04:46:53.533805    3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.530571    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.530706    3381 pod_workers.go:190] Error syncing pod 1c5609c5-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "392b8ee072b7bf80e79df45121440c891aaf8de230aa53964ef24b60bb92a776" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.531381    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-58lxc_kube-system(b7b01c12-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:46:54 minikube kubelet[3381]: E0410 04:46:54.531477    3381 pod_workers.go:190] Error syncing pod b7b01c12-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-58lxc_kube-system(b7b01c12-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "1fc8c7d721cae6017c0db69dc384eb0dcb08e35a0d7cf345ace875a6d3e9da33" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-58lxc_kube-system(b7b01c12-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:55 minikube kubelet[3381]: E0410 04:46:55.531816    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:46:55 minikube kubelet[3381]: E0410 04:46:55.531983    3381 pod_workers.go:190] Error syncing pod ef81cb54ca95dd4cf88bfbdf334b363b ("kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)"), skipping: failed to "ConfigPodSandbox" for "d296c674f6a728e34637935bb0b34325b87037d70c11eabb5d90a9265f7f60df" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:46:56 minikube kubelet[3381]: E0410 04:46:56.171467    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:46:56 minikube kubelet[3381]: E0410 04:46:56.174451    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:46:56 minikube kubelet[3381]: E0410 04:46:56.178389    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory

journalctl.logs

-- Logs begin at Wed 2019-04-10 04:38:40 UTC, end at Wed 2019-04-10 04:48:41 UTC. --
Apr 10 04:44:01 minikube kubelet[3381]: E0410 04:44:01.531469    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:01 minikube kubelet[3381]: E0410 04:44:01.531745    3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:05 minikube kernel: hpet1: lost 319 rtc interrupts
Apr 10 04:44:05 minikube kubelet[3381]: E0410 04:44:05.535691    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:05 minikube kubelet[3381]: E0410 04:44:05.536757    3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:06 minikube kubelet[3381]: E0410 04:44:06.172876    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:06 minikube kubelet[3381]: E0410 04:44:06.175689    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:06 minikube kubelet[3381]: E0410 04:44:06.181834    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:10 minikube kernel: hpet1: lost 317 rtc interrupts
Apr 10 04:44:10 minikube kubelet[3381]: E0410 04:44:10.529817    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:10 minikube kubelet[3381]: E0410 04:44:10.529861    3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.532928    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.533038    3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.533863    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.533918    3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877333    3381 remote_image.go:112] PullImage "ceph/ceph:v14.2.0-20190319" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877367    3381 kuberuntime_image.go:51] Pull image "ceph/ceph:v14.2.0-20190319" failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877456    3381 kuberuntime_manager.go:749] container start failed: ErrImagePull: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.877472    3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "StartContainer" for "version" with ErrImagePull: "rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.878328    3381 remote_image.go:112] PullImage "quay.io/k8scsi/csi-snapshotter:v1.0.1" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879367    3381 kuberuntime_image.go:51] Pull image "quay.io/k8scsi/csi-snapshotter:v1.0.1" failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879508    3381 kuberuntime_manager.go:749] container start failed: ErrImagePull: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879876    3381 remote_image.go:87] ImageStatus "quay.io/cephcsi/rbdplugin:v1.0.0" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879907    3381 kuberuntime_image.go:87] ImageStatus for image {"quay.io/cephcsi/rbdplugin:v1.0.0"} failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879938    3381 kuberuntime_manager.go:749] container start failed: ImageInspectError: Failed to inspect image "quay.io/cephcsi/rbdplugin:v1.0.0": rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879950    3381 remote_image.go:112] PullImage "quay.io/cephcsi/cephfsplugin:v1.0.0" from image service failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.879966    3381 kuberuntime_image.go:51] Pull image "quay.io/cephcsi/cephfsplugin:v1.0.0" failed: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.880010    3381 kuberuntime_manager.go:749] container start failed: ErrImagePull: rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.880729    3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: [failed to "StartContainer" for "csi-snapshotter" with ErrImagePull: "rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube kubelet[3381]: , failed to "StartContainer" for "csi-rbdplugin" with ImageInspectError: "Failed to inspect image \"quay.io/cephcsi/rbdplugin:v1.0.0\": rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube kubelet[3381]: ]
Apr 10 04:44:12 minikube kubelet[3381]: E0410 04:44:12.882122    3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "StartContainer" for "csi-cephfsplugin" with ErrImagePull: "rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/shim.sock" debug=false pid=9076
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12Z" level=info msg="shim reaped" id=8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12.987516753Z" level=error msg="stream copy error: reading from a closed fifo"
Apr 10 04:44:12 minikube dockerd[2399]: time="2019-04-10T04:44:12.987558237Z" level=error msg="stream copy error: reading from a closed fifo"
Apr 10 04:44:13 minikube dockerd[2399]: time="2019-04-10T04:44:13.011173432Z" level=error msg="8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a cleanup: failed to delete container from containerd: no such container"
Apr 10 04:44:13 minikube dockerd[2399]: time="2019-04-10T04:44:13.011222505Z" level=error msg="Handler for POST /v1.38/containers/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.012419    3381 remote_runtime.go:213] StartContainer "8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a" from runtime service failed: rpc error: code = Unknown desc = failed to start container "8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a": Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.012487    3381 kuberuntime_manager.go:749] container start failed: RunContainerError: failed to start container "8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a": Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.012509    3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "StartContainer" for "csi-cephfsplugin" with RunContainerError: "failed to start container \"8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a\": Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/8126fe0c8da3e5ffe46df5ca5014dabc83a8f6fefa442b7bc1e3d61c4659a09a/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.106412    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.106462    3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107038    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107065    3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107828    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.107858    3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.108944    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.109941    3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.530818    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:13 minikube kubelet[3381]: E0410 04:44:13.530911    3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:15 minikube kernel: hpet1: lost 320 rtc interrupts
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.177455    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.183538    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.189990    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.531697    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:16 minikube kubelet[3381]: E0410 04:44:16.531827    3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:20 minikube kernel: hpet1: lost 317 rtc interrupts
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.530788    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.530863    3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.530978    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.531023    3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.531825    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:24 minikube kubelet[3381]: E0410 04:44:24.531882    3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:25 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.530237    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.530391    3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.531630    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.531691    3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.532512    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:25 minikube kubelet[3381]: E0410 04:44:25.532565    3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.179720    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.207174    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.229088    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.531689    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:26 minikube kubelet[3381]: E0410 04:44:26.531840    3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:27 minikube audit: NETFILTER_CFG table=filter family=2 entries=41
Apr 10 04:44:27 minikube audit[9111]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d88881830 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube kernel: audit: type=1325 audit(1554871467.238:86): table=filter family=2 entries=41
Apr 10 04:44:27 minikube kernel: audit: type=1300 audit(1554871467.238:86): arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d88881830 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube audit: PROCTITLE proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube kernel: audit: type=1327 audit(1554871467.238:86): proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube audit: NETFILTER_CFG table=nat family=2 entries=62
Apr 10 04:44:27 minikube audit[9111]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d888855f0 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube kernel: audit: type=1325 audit(1554871467.242:87): table=nat family=2 entries=62
Apr 10 04:44:27 minikube kernel: audit: type=1300 audit(1554871467.242:87): arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=555d888855f0 items=0 ppid=4814 pid=9111 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:27 minikube audit: PROCTITLE proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube kernel: audit: type=1327 audit(1554871467.242:87): proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:27 minikube kubelet[3381]: E0410 04:44:27.533658    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:27 minikube kubelet[3381]: E0410 04:44:27.533779    3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:28 minikube kubelet[3381]: E0410 04:44:28.530741    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:28 minikube kubelet[3381]: E0410 04:44:28.530917    3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:30 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:30 minikube kubelet[3381]: E0410 04:44:30.531959    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:30 minikube kubelet[3381]: E0410 04:44:30.532120    3381 pod_workers.go:190] Error syncing pod 1c3c30c1-5b4b-11e9-ac49-08002742bbfb ("rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "d459439f566c6bb3db20d21694b6988ba9fd42d0b62a4e3942d9c41227773a6b" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.530760    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.530916    3381 pod_workers.go:190] Error syncing pod 17eea6fd9342634d7d40a04d577641fd ("kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)"), skipping: failed to "ConfigPodSandbox" for "146305c699f6d8b8260bce39b88f656fb86ca3043b90c653f8f3fb275ab16c85" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.532209    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:32 minikube kubelet[3381]: E0410 04:44:32.532323    3381 pod_workers.go:190] Error syncing pod 1c5609c5-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "392b8ee072b7bf80e79df45121440c891aaf8de230aa53964ef24b60bb92a776" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:35 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.174051    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.177904    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.187510    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.530998    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.531070    3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.531773    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.531809    3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.532157    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:36 minikube kubelet[3381]: E0410 04:44:36.532203    3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:37 minikube kubelet[3381]: E0410 04:44:37.531047    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:37 minikube kubelet[3381]: E0410 04:44:37.531176    3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:38 minikube kubelet[3381]: E0410 04:44:38.535568    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:38 minikube kubelet[3381]: E0410 04:44:38.535734    3381 pod_workers.go:190] Error syncing pod 4b52d75cab61380f07c0c5a69fb371d4 ("kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)"), skipping: failed to "ConfigPodSandbox" for "01457429d7542320cf53936ab4023e78fb08cadccd55dde9412ea53ae5161936" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.554986    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.555594    3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.555118    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.555737    3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.556946    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:39 minikube kubelet[3381]: E0410 04:44:39.557064    3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:40 minikube kernel: hpet1: lost 318 rtc interrupts
Apr 10 04:44:42 minikube kubelet[3381]: E0410 04:44:42.539856    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:42 minikube kubelet[3381]: E0410 04:44:42.540295    3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:43 minikube kubelet[3381]: E0410 04:44:43.537196    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:43 minikube kubelet[3381]: E0410 04:44:43.537281    3381 pod_workers.go:190] Error syncing pod 1c5a8cbb-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "4c6f8fc54c8b91c59ae60c70b073dfe73939e4bd83344106feffd315661d2473" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-cg764_smoke-ns-system(1c5a8cbb-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:45 minikube kernel: hpet1: lost 320 rtc interrupts
Apr 10 04:44:45 minikube kubelet[3381]: E0410 04:44:45.530651    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:45 minikube kubelet[3381]: E0410 04:44:45.530799    3381 pod_workers.go:190] Error syncing pod 1c3c30c1-5b4b-11e9-ac49-08002742bbfb ("rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "d459439f566c6bb3db20d21694b6988ba9fd42d0b62a4e3942d9c41227773a6b" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.177820    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.189574    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.198624    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.530807    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:46 minikube kubelet[3381]: E0410 04:44:46.530952    3381 pod_workers.go:190] Error syncing pod 17eea6fd9342634d7d40a04d577641fd ("kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)"), skipping: failed to "ConfigPodSandbox" for "146305c699f6d8b8260bce39b88f656fb86ca3043b90c653f8f3fb275ab16c85" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-controller-manager-minikube_kube-system(17eea6fd9342634d7d40a04d577641fd)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:47 minikube kubelet[3381]: E0410 04:44:47.530410    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:47 minikube kubelet[3381]: E0410 04:44:47.530630    3381 pod_workers.go:190] Error syncing pod 1c5609c5-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "392b8ee072b7bf80e79df45121440c891aaf8de230aa53964ef24b60bb92a776" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-attacher-0_smoke-ns-system(1c5609c5-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.532267    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.532593    3381 pod_workers.go:190] Error syncing pod 1c4d6f79-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "a190563612f0e5cf3035ba8192209aac99b53639658f6a460f69207e696cef59" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-provisioner-0_smoke-ns-system(1c4d6f79-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.533392    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:48 minikube kubelet[3381]: E0410 04:44:48.533490    3381 pod_workers.go:190] Error syncing pod 1afbd371-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "be1ed8b86f84208e51f540c3007bcfdad491fc13f2b495276ca536cd800223a3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-operator-7f5858cb-ngrd5_smoke-ns-system(1afbd371-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:49 minikube kubelet[3381]: E0410 04:44:49.536147    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:49 minikube kubelet[3381]: E0410 04:44:49.536487    3381 pod_workers.go:190] Error syncing pod b84450fa-5b4a-11e9-ac49-08002742bbfb ("storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "37bf0aa77e594d17d0c4d59678265713bdc73d5c84cd4be305e7a51bd38d4eeb" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"storage-provisioner_kube-system(b84450fa-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:50 minikube kernel: hpet1: lost 317 rtc interrupts
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.534434    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.534826    3381 pod_workers.go:190] Error syncing pod 1c3a0bb6-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "fe4fdce7a437b5a6d8f7973a695632d75fe12959520336b2eb41a32d5e6fe7e3" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-agent-kqpzk_smoke-ns-system(1c3a0bb6-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.536539    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:50 minikube kubelet[3381]: E0410 04:44:50.536712    3381 pod_workers.go:190] Error syncing pod 1c61e899-5b4b-11e9-ac49-08002742bbfb ("csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "c100927f5d927fe9ef5b5c14aec9dc9059d818d91aae56d99189811dacb1ffe4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-cephfsplugin-provisioner-0_smoke-ns-system(1c61e899-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.545519    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.546282    3381 pod_workers.go:190] Error syncing pod 2346a0b2-5b4b-11e9-ac49-08002742bbfb ("rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "df05f51697b0bae538debec7ba0dc66664485bf2d08b3df84678efcc0a76bbb1" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-ceph-detect-version-mkvzj_smoke-ns(2346a0b2-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.557738    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:51 minikube kubelet[3381]: E0410 04:44:51.557931    3381 pod_workers.go:190] Error syncing pod ef81cb54ca95dd4cf88bfbdf334b363b ("kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)"), skipping: failed to "ConfigPodSandbox" for "d296c674f6a728e34637935bb0b34325b87037d70c11eabb5d90a9265f7f60df" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-addon-manager-minikube_kube-system(ef81cb54ca95dd4cf88bfbdf334b363b)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.535959    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.536385    3381 pod_workers.go:190] Error syncing pod 4b52d75cab61380f07c0c5a69fb371d4 ("kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)"), skipping: failed to "ConfigPodSandbox" for "01457429d7542320cf53936ab4023e78fb08cadccd55dde9412ea53ae5161936" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-scheduler-minikube_kube-system(4b52d75cab61380f07c0c5a69fb371d4)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.539699    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "kube-proxy-gts8g_kube-system(b7e6760f-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:53 minikube kubelet[3381]: E0410 04:44:53.540229    3381 pod_workers.go:190] Error syncing pod b7e6760f-5b4a-11e9-ac49-08002742bbfb ("kube-proxy-gts8g_kube-system(b7e6760f-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "8dccb241058a7c51d45d93f4a6f58d268d2d8a7b8b6dd9f44a2a24b6000b75b4" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"kube-proxy-gts8g_kube-system(b7e6760f-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.530768    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.530873    3381 pod_workers.go:190] Error syncing pod 1c45ab18-5b4b-11e9-ac49-08002742bbfb ("csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "10c39abb63b7f6489c04354072eca6b9dee7f2b1c3c48edefb871499fa780fb8" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"csi-rbdplugin-vw4ch_smoke-ns-system(1c45ab18-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.531639    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:54 minikube kubelet[3381]: E0410 04:44:54.532034    3381 pod_workers.go:190] Error syncing pod b7af6cca-5b4a-11e9-ac49-08002742bbfb ("coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "744911a30bb0370b176f2579eef009177a1a33657396d5739c8eae67ce5bf414" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"coredns-86c58d9df4-t8hhp_kube-system(b7af6cca-5b4a-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:55 minikube kernel: hpet1: lost 319 rtc interrupts
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.174345    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.187741    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.194960    3381 remote_runtime.go:336] ExecSync d113d9b6c6e595b392ee48295a0384e6bdf53cff5e72a81dd8cce0c0b4e0f14e '/bin/sh -ec ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/var/lib/minikube/certs//etcd/ca.crt --cert=/var/lib/minikube/certs//etcd/healthcheck-client.crt --key=/var/lib/minikube/certs//etcd/healthcheck-client.key get foo' from runtime service failed: rpc error: code = Unknown desc = cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.531670    3381 kuberuntime_manager.go:699] GeneratePodSandboxConfig for pod "rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)" failed: open /etc/resolv.conf: no such file or directory
Apr 10 04:44:56 minikube kubelet[3381]: E0410 04:44:56.532533    3381 pod_workers.go:190] Error syncing pod 1c3c30c1-5b4b-11e9-ac49-08002742bbfb ("rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)"), skipping: failed to "ConfigPodSandbox" for "d459439f566c6bb3db20d21694b6988ba9fd42d0b62a4e3942d9c41227773a6b" with ConfigPodSandboxError: "GeneratePodSandboxConfig for pod \"rook-discover-dqdjh_smoke-ns-system(1c3c30c1-5b4b-11e9-ac49-08002742bbfb)\" failed: open /etc/resolv.conf: no such file or directory"
Apr 10 04:44:57 minikube audit: NETFILTER_CFG table=filter family=2 entries=41
Apr 10 04:44:57 minikube audit[9177]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=5608a3e43830 items=0 ppid=4814 pid=9177 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:57 minikube audit: PROCTITLE proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:57 minikube kernel: audit: type=1325 audit(1554871497.289:88): table=filter family=2 entries=41
Apr 10 04:44:57 minikube kernel: audit: type=1300 audit(1554871497.289:88): arch=c000003e syscall=54 success=yes exit=0 a0=3 a1=0 a2=40 a3=5608a3e43830 items=0 ppid=4814 pid=9177 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/sbin/xtables-multi" subj=kernel key=(null)
Apr 10 04:44:57 minikube kernel: audit: type=1327 audit(1554871497.289:88): proctitle=69707461626C65732D726573746F7265002D2D6E6F666C757368002D2D636F756E74657273
Apr 10 04:44:57 minikube audit: NETFILTER_CFG table=nat family=2 entries=62

The text was updated successfully, but these errors were encountered:

tstromberg · 2019-04-11T18:01:03Z

I'm not yet familiar with bi-directional mounts. Any chance you could post instructions to reproduce this bug?

Madhu-1 · 2019-05-09T03:48:52Z

https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/csi/template/cephfs/csi-cephfsplugin.yaml#L71 this is the bi-directional mount used in the pod template
the full template is here https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/csi/template/cephfs/csi-cephfsplugin.yaml

kfox1111 · 2019-06-20T16:14:10Z

I'm seeing something similar maybe. I fired up minikube 1.1.1 and tried to load in a csi driver of my own and odd things start happening immediately after the driver registers. containers fail to start complaining /etc/resolv.conf doesn't exist anymore. logging onto the host, and catting it, says it doesn't exist either. very strange. I confirmed it doesn't do this outside of minikube. Should I file this as a separate issue, or keep it as part of this one?

giovanism · 2019-06-30T10:31:29Z

@Madhu-1 Have you enabled the mount propagation?

@tstromberg , Is there any way to pass MountFlags=shared to docker systemd unit file inside minikube?

kfox1111 · 2019-07-03T16:35:37Z

Still seeing this with minikube 1.2.0.
Reproducer:
minikube start --feature-gates CSIInlineVolume=true
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-image-populator/master/deploy/image/csi-image-csidriverinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-image-populator/master/deploy/image/csi-image-daemonset.yaml

kubectl get pods #shows the pod never fully starting. You start to get weird errors like:
minikube ssh cat /etc/resolv.conf
cat: /etc/resolv.conf: No such file or directory
ssh: Process exited with status 1

kfox1111 · 2019-07-09T22:50:39Z

minikube ssh
sudo su -
mkdir -p /etc/systemd/system/docker.service.d/
(echo '[Service]'; echo 'MountFlags=shared') > /etc/systemd/system/docker.service.d/mount_flags.conf
 systemctl daemon-reload
systemctl restart docker
systemctl restart kubelet
systemctl show docker | grep Mount
#MountFlags=shared
exit
exit
#no workie
minikube stop
minikube start
#no workie
#/etc/systemd/system/docker.service.d/mount_flags.conf is missing after reboot.

kfox1111 · 2019-07-09T22:53:25Z

Ah.... I found this: opencontainers/runc#1961
inspecting, it looks like it is the issue:

# mount | grep root
rootfs on / type rootfs (rw)
# df -h /
Filesystem      Size  Used Avail Use% Mounted on
overlay            0     0     0    - /
#

kfox1111 · 2019-07-09T23:17:06Z

I can confirm it with cri-o too:

minikube start -p crio --container-runtime=cri-o 
#confirm that it is cri-o based
minikube -p crio ssh ps ax | grep crio
 1988 ?        Ssl    0:13 /usr/bin/crio --log-level=debug --insecure-registry 10.96.0.0/12
 2860 ?        Ssl    0:00 /usr/libexec/crio/conmon --syslog -c 4e35107dd63ff5eef5b0adf9d6eb442f6a88429c40ce702523dae2a22ea99bdf -n k8s_POD_kube-addon-manager-minikube_kube-system_65a31d2b812b11a2035f37c8a742e46
 2891 ?        Ssl    0:00 /usr/libexec/crio/conmon --syslog -c 085c98ce9552d32c5d8d0966a72685fd6753e547ddbea7f9d93acd3d04ac8b54 -n k8s_kube-addon-manager_kube-addon-manager-minikube_kube-system_65a31d2b812b11a2
#check the mount
minikube -p crio ssh -- df -h /
Filesystem      Size  Used Avail Use% Mounted on
rootfs             0     0     0    - /
#apply the driver
kubectl apply -f git/csi-driver-image-populator/deploy/image/csi-image-daemonset.yaml
#wait a bit for it to sort of start/crash
minikube -p crio ssh -- df -h /
Filesystem      Size  Used Avail Use% Mounted on
overlay            0     0     0    - /

kfox1111 · 2019-07-18T15:58:27Z

Is there any way to switch the root filesystem away from rootfs?

ryysud · 2019-08-09T08:10:33Z

I faced the same problem (csi driver not working) with kubernetes-sigs/secrets-store-csi-driver#49 .

entropitor · 2019-08-12T16:13:35Z

Is there anything we can do to help with this issue? Provide more context / ... Basically CSI is not working for minikube which seems like a big deal (given minikube tries to be a complete kubernetes setup)?

kfox1111 · 2019-08-14T00:08:15Z

More debugging in #3512
Looks like its not necessarily related to rootfs.

Its very odd. Its like mounts intended for the containers are leaking back out to the host.

kfox1111 · 2019-08-14T00:29:11Z

Comparing mounts before and after, it looks like its duplicating a lot of mounts... something really odd is happening.

kfox1111 · 2019-08-14T00:42:30Z

So, I tried looking at a box with working propagation:

findmnt -o TARGET,PROPAGATION | grep -v shared
TARGET                                PROPAGATION
├─/var/lib/docker/containers          private
└─/var/lib/docker/overlay2            private

It has two private mounts. Tried this on minikube:

mount --bind /var/lib/docker /var/lib/docker
mount --make-private /var/lib/docker
mount --bind /var/lib/docker/containers /var/lib/docker/containers
mount --make-private /var/lib/docker/containers
mount --bind /var/lib/docker/overlay2 /var/lib/docker/overlay2
mount --make-private /var/lib/docker/overlay2

No joy.

Container mounts are still leaking out to the host:

$ mount | grep resolv
/dev/vda1 on /etc/resolv.conf type ext4 (rw,relatime,data=ordered)

kfox1111 · 2019-08-14T16:26:57Z

Ok. I got it to work!
I had to also remove Environment=DOCKER_RAMDISK=yes from /usr/lib/systemd/system/docker.service

Next I need to unwind a bit and see which of the things above are really required and which are not.

leseb · 2019-08-20T13:25:55Z

@kfox1111 how do you override /usr/lib/systemd/system/docker.service, DOCKER_RAMDISK=yes comes back after reboot for me.

Thanks.

kfox1111 · 2019-08-20T16:00:51Z

@kfox1111 how do you override /usr/lib/systemd/system/docker.service, DOCKER_RAMDISK=yes comes back after reboot for me.

I shutdown the vm. I then manually started it in libvirt, attached to the console, logged in. I edited /usr/lib/systemd/system/docker.service. Then systemctl daemon-reload; systemctl restart docker;

Then you: minikube start
It will overwrite the docker.service, but since docker is already running it skips starting it. Everything then came up ok.

JanMatas · 2019-08-23T12:40:17Z

Hi,

I am experiencing the same issue, but I am afraid I am abit confused by the thread ^ . Is there any chance someone could summarize the workaround?

kfox1111 · 2019-08-23T16:42:29Z

Its fairly complicated to get the fix in place. You need to extract the iso, extract the initrd from it, update the init script, then rebuild the initrd, and then rebuild the iso from it. overwrite the iso from a previously created minikube, boot the vm not from minikube start but manually, login through the console, update docker.service to remove the environment variable, systemctl daemon-reload and restart the docker service, then minikube start.

The fix should be making it into minikube 1.4.0 so it may be easier to wait unless you really need it now.

doprdele · 2019-08-28T17:05:22Z

Any ETA on Minikube 1.4.0? How do you know the fix is making it into 1.4.0 @kfox1111 ?

afbjorklund · 2019-08-28T17:46:36Z

There will be a beta ISO out soon

kfox1111 · 2019-08-28T17:55:23Z

@doprdele #5133 merged.

tstromberg · 2019-09-20T04:17:54Z

@madhu-1I believe this issue is now addressed by minikube v1.4, as it the necessary fixes have been merged. If you still see this issue with minikube v1.4 or higher, please reopen this issue by commenting with /reopen

Thank you for reporting this issue!

Lanayx · 2019-11-29T09:45:06Z

/reopen

k8s-ci-robot · 2019-11-29T09:45:13Z

@Lanayx: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Lanayx · 2019-11-29T09:47:04Z

Hi, I'm getting the same symptoms with minikube 1.5.2 on windows+hyperv, when trying to install rook cluster

Lanayx · 2019-11-30T14:01:50Z

With minikube-v1.6.0-beta.1 it works for me

Madhu-1 mentioned this issue Apr 10, 2019

Fix driver name in ceph csi driver rook/rook#2839

Merged

5 tasks

tstromberg added priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. triage/needs-information Indicates an issue needs more information in order to work on it. co/docker-env docker-env issues labels Apr 11, 2019

tstromberg added r/2019q2 Issue was last reviewed 2019q2 and removed triage/needs-information Indicates an issue needs more information in order to work on it. labels May 23, 2019

kfox1111 mentioned this issue Jul 18, 2019

Migrate away from rootfs / DOCKER_RAMDISK #3512

Closed

kfox1111 mentioned this issue Jul 31, 2019

PVC/PV conformance testing discussion kubernetes/kubernetes#65155

Open

ryysud mentioned this issue Aug 9, 2019

The Secrets Store CSI Driver installation fails on minikube kubernetes-sigs/secrets-store-csi-driver#49

Closed

leseb mentioned this issue Aug 20, 2019

Move root filesystem from rootfs to tmpfs #5133

Merged

tstromberg closed this as completed Sep 20, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enabling bidirectional mounts on pod kills docker #4072

Enabling bidirectional mounts on pod kills docker #4072

Madhu-1 commented Apr 10, 2019

tstromberg commented Apr 11, 2019

Madhu-1 commented May 9, 2019

kfox1111 commented Jun 20, 2019

giovanism commented Jun 30, 2019 •

edited

Loading

kfox1111 commented Jul 3, 2019

kfox1111 commented Jul 9, 2019

kfox1111 commented Jul 9, 2019

kfox1111 commented Jul 9, 2019

kfox1111 commented Jul 18, 2019

ryysud commented Aug 9, 2019

entropitor commented Aug 12, 2019

kfox1111 commented Aug 14, 2019

kfox1111 commented Aug 14, 2019

kfox1111 commented Aug 14, 2019

kfox1111 commented Aug 14, 2019

leseb commented Aug 20, 2019

kfox1111 commented Aug 20, 2019

JanMatas commented Aug 23, 2019

kfox1111 commented Aug 23, 2019

doprdele commented Aug 28, 2019

afbjorklund commented Aug 28, 2019

kfox1111 commented Aug 28, 2019

tstromberg commented Sep 20, 2019

Lanayx commented Nov 29, 2019

k8s-ci-robot commented Nov 29, 2019

Lanayx commented Nov 29, 2019

Lanayx commented Nov 30, 2019

Enabling bidirectional mounts on pod kills docker #4072

Enabling bidirectional mounts on pod kills docker #4072

Comments

Madhu-1 commented Apr 10, 2019

tstromberg commented Apr 11, 2019

Madhu-1 commented May 9, 2019

kfox1111 commented Jun 20, 2019

giovanism commented Jun 30, 2019 • edited Loading

kfox1111 commented Jul 3, 2019

kfox1111 commented Jul 9, 2019

kfox1111 commented Jul 9, 2019

kfox1111 commented Jul 9, 2019

kfox1111 commented Jul 18, 2019

ryysud commented Aug 9, 2019

entropitor commented Aug 12, 2019

kfox1111 commented Aug 14, 2019

kfox1111 commented Aug 14, 2019

kfox1111 commented Aug 14, 2019

kfox1111 commented Aug 14, 2019

leseb commented Aug 20, 2019

kfox1111 commented Aug 20, 2019

JanMatas commented Aug 23, 2019

kfox1111 commented Aug 23, 2019

doprdele commented Aug 28, 2019

afbjorklund commented Aug 28, 2019

kfox1111 commented Aug 28, 2019

tstromberg commented Sep 20, 2019

Lanayx commented Nov 29, 2019

k8s-ci-robot commented Nov 29, 2019

Lanayx commented Nov 29, 2019

Lanayx commented Nov 30, 2019

giovanism commented Jun 30, 2019 •

edited

Loading