jiva volume status shows unknown upon continuous restart of replica in a specific node

[nsathyaseelan]

What steps did you take and what happened:

• Continuously restarting the replica that was scheduled on the specific node post that checking the replica status in jivavolume not able to get the status of the volume.

d2iq@rack2:~/e2e-konvoy/openebs-konvoy-e2e/stages/5-infra-chaos$ kubectl get jivavolume -n openebs
NAME                                       REPLICACOUNT   PHASE     STATUS
pvc-4870269c-2423-4279-8be9-978d68ac59ac   3              Ready     RW
pvc-5180e0ee-61fb-4685-bb95-a0f3c695032d   3              Ready     RW
pvc-b67872c9-0e9e-4073-b346-2fde70031600                  Unknown  

d2iq@rack2:~/e2e-konvoy/openebs-konvoy-e2e/stages/5-infra-chaos$ kubectl get po -n openebs  -o wide
NAME                                                              READY   STATUS    RESTARTS   AGE     IP                NODE         NOMINATED NODE   READINESS GATES
containerd-chaos-69cvc                                            1/1     Running   0          29m     192.168.22.156    d2iq-node2   <none>           <none>
containerd-chaos-gcdm8                                            1/1     Running   1          29m     192.168.49.244    d2iq-node5   <none>           <none>
containerd-chaos-jsd58                                            1/1     Running   0          29m     192.168.42.188    d2iq-node3   <none>           <none>
containerd-chaos-l6spd                                            1/1     Running   0          29m     192.168.144.238   d2iq-node1   <none>           <none>
containerd-chaos-xjjzl                                            1/1     Running   0          29m     192.168.205.238   d2iq-node4   <none>           <none>
jiva-operator-5b8b6d668f-lb8v6                                    1/1     Running   1          46m     192.168.22.145    d2iq-node2   <none>           <none>
maya-apiserver-bcfb957d7-gml5s                                    1/1     Running   0          50m     192.168.144.204   d2iq-node1   <none>           <none>
openebs-admission-server-75b69bc554-vgw24                         1/1     Running   0          50m     192.168.205.206   d2iq-node4   <none>           <none>
openebs-jiva-csi-controller-0                                     5/5     Running   0          46m     192.168.205.208   d2iq-node4   <none>           <none>
openebs-jiva-csi-node-6pmnp                                       3/3     Running   4          46m     10.43.1.115       d2iq-node5   <none>           <none>
openebs-jiva-csi-node-6w688                                       3/3     Running   0          46m     10.43.1.113       d2iq-node3   <none>           <none>
openebs-jiva-csi-node-99lb6                                       3/3     Running   0          46m     10.43.1.111       d2iq-node1   <none>           <none>
openebs-jiva-csi-node-gf84p                                       3/3     Running   0          46m     10.43.1.112       d2iq-node2   <none>           <none>
openebs-jiva-csi-node-rkdtr                                       3/3     Running   0          46m     10.43.1.114       d2iq-node4   <none>           <none>
openebs-localpv-provisioner-864958cb7b-8wpfg                      1/1     Running   0          50m     192.168.144.206   d2iq-node1   <none>           <none>
openebs-ndm-44tlh                                                 1/1     Running   0          47m     10.43.1.111       d2iq-node1   <none>           <none>
openebs-ndm-djw9p                                                 1/1     Running   0          47m     10.43.1.112       d2iq-node2   <none>           <none>
openebs-ndm-operator-6f4bc6c84d-dcmvs                             1/1     Running   0          48m     192.168.144.207   d2iq-node1   <none>           <none>
openebs-ndm-qkct4                                                 1/1     Running   0          47m     10.43.1.114       d2iq-node4   <none>           <none>
openebs-ndm-smj5f                                                 1/1     Running   2          47m     10.43.1.115       d2iq-node5   <none>           <none>
openebs-ndm-wgsj5                                                 1/1     Running   0          47m     10.43.1.113       d2iq-node3   <none>           <none>
openebs-provisioner-cff4c4454-nvtfb                               1/1     Running   0          50m     192.168.205.205   d2iq-node4   <none>           <none>
openebs-snapshot-operator-6c5d68548c-7c9nc                        2/2     Running   0          50m     192.168.22.144    d2iq-node2   <none>           <none>
pvc-4870269c-2423-4279-8be9-978d68ac59ac-jiva-ctrl-787f746729f9   1/1     Running   0          41m     192.168.42.148    d2iq-node3   <none>           <none>
pvc-4870269c-2423-4279-8be9-978d68ac59ac-jiva-rep-0               1/1     Running   0          41m     192.168.42.150    d2iq-node3   <none>           <none>
pvc-4870269c-2423-4279-8be9-978d68ac59ac-jiva-rep-1               1/1     Running   2          41m     192.168.22.154    d2iq-node2   <none>           <none>
pvc-4870269c-2423-4279-8be9-978d68ac59ac-jiva-rep-2               1/1     Running   1          41m     192.168.49.236    d2iq-node5   <none>           <none>
pvc-4d85b5c2-f526-431d-a24d-8b450dfbcf07-jiva-ctrl-7b4dd699fbpg   1/1     Running   1          8m15s   192.168.49.250    d2iq-node5   <none>           <none>
pvc-4d85b5c2-f526-431d-a24d-8b450dfbcf07-jiva-rep-0               1/1     Running   3          8m15s   192.168.22.170    d2iq-node2   <none>           <none>
pvc-4d85b5c2-f526-431d-a24d-8b450dfbcf07-jiva-rep-1               1/1     Running   1          8m15s   192.168.49.195    d2iq-node5   <none>           <none>
pvc-4d85b5c2-f526-431d-a24d-8b450dfbcf07-jiva-rep-2               1/1     Running   4          8m15s   192.168.42.136    d2iq-node3   <none>           <none>
pvc-5180e0ee-61fb-4685-bb95-a0f3c695032d-jiva-ctrl-5f5fbfdcnq6n   1/1     Running   0          40m     192.168.42.160    d2iq-node3   <none>           <none>
pvc-5180e0ee-61fb-4685-bb95-a0f3c695032d-jiva-rep-0               1/1     Running   0          40m     192.168.42.164    d2iq-node3   <none>           <none>
pvc-5180e0ee-61fb-4685-bb95-a0f3c695032d-jiva-rep-1               1/1     Running   0          40m     192.168.205.212   d2iq-node4   <none>           <none>
pvc-5180e0ee-61fb-4685-bb95-a0f3c695032d-jiva-rep-2               1/1     Running   0          40m     192.168.144.220   d2iq-node1   <none>           <none>
pvc-b67872c9-0e9e-4073-b346-2fde70031600-jiva-ctrl-cd54f5br7txm   1/1     Running   1          32m     192.168.49.249    d2iq-node5   <none>           <none>
pvc-b67872c9-0e9e-4073-b346-2fde70031600-jiva-rep-0               1/1     Running   4          26m     192.168.49.239    d2iq-node5   <none>           <none>
pvc-b67872c9-0e9e-4073-b346-2fde70031600-jiva-rep-1               1/1     Running   1          32m     192.168.42.184    d2iq-node3   <none>           <none>
pvc-b67872c9-0e9e-4073-b346-2fde70031600-jiva-rep-2               1/1     Running   2          32m     192.168.22.130    d2iq-node2   <none>           <none>

What did you expect to happen:

• Jivavolume status should be RW and ready phase.

jv-oyaml.txt
current_ctrl_pod.log
controller-pod.log
jiva-operator.log