openEBS ISCSI Error

[segator]

RancherOS Version: (ros os version)
1.4.0
Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)
vsphere 6.5

Running rancher2
OpenEBS from helm

I got this error
MountVolume.WaitForAttach failed for volume "pvc-5cc527d9-9a71-11e8-88ab-005056a69a74" : failed to get any path for iscsi disk, last err seen: iscsi: failed to sendtargets to portal 10.43.169.43:3260 output: iscsiadm: Could not open /run/lock/iscsi: No such file or directory iscsiadm: Could not open /run/lock/iscsi: No such file or directory iscsiadm: Could not open /run/lock/iscsi: No such file or directory iscsiadm: Could not add new discovery record. , err exit status 6 2 minutes ago

yes I executed ros service up open-iscsi

if I open a ssh connection to the VM and I execute command iscsid automatically openEBS volumes work
but If I execute ps aux | grep iscsid the new process apear as detached

Before / After
root@isaac-master1:/home/rancher# ps aux | grep iscsi root 6337 0.0 0.0 0 0 ? I< Aug06 0:00 [iscsi_eh] root 21170 0.0 0.0 24984 284 ? Ss 18:50 0:00 /sbin/iscsid root 21171 0.0 0.1 33928 5308 ? S<Ls 18:50 0:00 /sbin/iscsid root 25273 0.0 0.0 11284 964 pts/0 S+ 18:56 0:00 grep --color=auto iscsi root@isaac-master1:/home/rancher# iscsid root@isaac-master1:/home/rancher# ps aux | grep iscsi root 6337 0.0 0.0 0 0 ? I< Aug06 0:00 [iscsi_eh] root 21170 0.0 0.0 24984 284 ? Ss 18:50 0:00 /sbin/iscsid root 21171 0.0 0.1 33928 5308 ? S<Ls 18:50 0:00 /sbin/iscsid root 25331 0.0 0.0 0 0 ? Zs 18:56 0:00 [iscsid] <defunct> root 25332 0.0 0.0 0 0 ? Z 18:56 0:00 [iscsid] <defunct> root 25387 0.0 0.0 11284 976 pts/0 S+ 18:56 0:00 grep --color=auto iscsi

[niusmallnan]

I recommend you use the ubuntu console:

$ ros console switch ubuntu
$ apt update
$ apt install --no-install-recommends open-iscsi

You need to create a lock directory, and make sure to create this directory every boot:

$ mkdir /run/lock

# update cloud-config
#cloud-config
runcmd:
- [mkdir, /run/lock]

[segator]

Can I add the apt install --no-install-recommends open-iscsi inside the /opt/rancher/bin/start.sh?

runcmd is executed before or after /etc/rc.local and before or after /opt/rancher/bin/start.sh?

thank you!

[niusmallnan]

runcmd ---> /opt/rancher/bin/start.sh ----> /etc/rc.local

[segator]

It seems the error is gone but now appears another

MountVolume.WaitForAttach failed for volume "pvc-e36b99e9-9cd8-11e8-b3c0-005056a61abf" : failed to get any path for iscsi disk, last err seen: iscsi: failed to sendtargets to portal 10.43.16.186:3260 output: iscsiadm: connect to 10.43.16.186 timed out iscsiadm: connect to 10.43.16.186 timed out iscsiadm: connect to 10.43.16.186 timed out iscsiadm: connect to 10.43.16.186 timed out iscsiadm: connect to 10.43.16.186 timed out iscsiadm: connect to 10.43.16.186 timed out iscsiadm: connection login retries (reopen_max) 5 exceeded iscsiadm: No portals found , err exit status 21

[niusmallnan]

If Ubuntu OS works fine with your case, I will check it out if there is a problem with RancherOS.
Otherwise, I am not sure if it is a problem with RancherOS or a problem with setup configuration.

[segator]

Well the setup is so simple.

rancher server create cluster using vsphere and loading custom cloud-init
Network tried with flannel and calico(same error)
The cloud init install company CA Certs and open-iscsi.

Then deploy openEBS from rancher catalog
Load default storage class https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-storageclasses.yaml

Then try to deploy a service with PVC on any storage class of openebs.

[niusmallnan]

@rootwuj Please help to confirm this issue when you have time. It's better to confirm if openebs(from rancher catalog) can work on Ubuntu.
ping @kingsd041

[rootwuj]

I added RKE cluster of Ubuntu OS on Rancher2 (the network is flannel), Then deploy openEBS from rancher catalog, and finally created the PVC using the storage class of the openebs. PVC was created successfully.

[segator]

well the problem is related with rancherOS not Ubuntu

[tibers]

I came here from google...

$ kubectl describe pod dbench-2knzm --namespace dbench-openebs
...
Events:
  Type     Reason                  Age               From                      Message
  ----     ------                  ----              ----                      -------
  Normal   Scheduled               57s               default-scheduler         Successfully assigned dbench-openebs/dbench-2knzm to node10173119244
  Normal   SuccessfulAttachVolume  57s               attachdetach-controller   AttachVolume.Attach succeeded for volume "pvc-524e1b80-dbd6-11e8-b0b9-12ad5e99cc8f"
  Warning  FailedMount             8s (x7 over 40s)  kubelet, node10173119244  MountVolume.WaitForAttach failed for volume "pvc-524e1b80-dbd6-11e8-b0b9-12ad5e99cc8f" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to sendtargets to portal 10.43.172.115:3260 output: iscsiadm: Could not open /run/lock/iscsi: No such file or directory
iscsiadm: Could not open /run/lock/iscsi: No such file or directory
iscsiadm: Could not open /run/lock/iscsi: No such file or directory
iscsiadm: Could not add new discovery record.
, err exit status 6

But I don't know how to fix it. Stop using rancherOS?

NINJA EDIT: I followed the commands here and it works.

[niusmallnan]

I will make dir /run/lock automatically in the console so that you don't have to configure runcmd.

If someone is running kubelet in a container, one thing to note： https://docs.openebs.io/docs/next/tsgiscsi.html
Rancher uses RKE to build the k8s cluster, and RKE runs kubelet in a container.

[rootwuj]

Tested with RancherOS version- v1.5.0-rc2(Kernel Version: 4.14.85-rancher) with AWS.
This version of OpenEBS can be automatically made dir /run/lock.